Oct 29 22:12:04.595239 ip-10-0-134-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 29 22:12:04.595252 ip-10-0-134-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 29 22:12:04.595263 ip-10-0-134-94 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 29 22:12:04.595760 ip-10-0-134-94 systemd[1]: Failed to start Kubernetes Kubelet. Oct 29 22:12:14.413319 ip-10-0-134-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Oct 29 22:12:14.413335 ip-10-0-134-94 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6e09bfe67c0e464db358fdcb2e66c23e -- Oct 29 22:14:02.655464 ip-10-0-134-94 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:14:03.105814 ip-10-0-134-94 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:03.105814 ip-10-0-134-94 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:14:03.105814 ip-10-0-134-94 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:03.105814 ip-10-0-134-94 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:03.112051 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.111940 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.112051 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112053 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112059 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112062 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112065 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112069 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112072 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112075 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112078 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112081 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112093 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112098 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112100 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112104 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112106 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112109 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112111 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112114 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112117 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112120 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112122 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.112127 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112125 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112128 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112130 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112133 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112137 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112139 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112165 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112168 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112171 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112173 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112176 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112178 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112181 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112183 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112186 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112188 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112191 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112193 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.112600 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112198 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112202 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112204 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112208 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112211 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112214 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112216 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112225 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112227 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112230 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112233 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112238 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112241 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112244 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112247 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112249 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112252 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112255 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112257 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.113042 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112260 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112262 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112265 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112267 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112270 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112272 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112275 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112277 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112280 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112282 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112285 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112287 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112290 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112292 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112294 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112297 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112300 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112302 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112306 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112308 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.113519 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112318 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112321 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112325 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112328 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112331 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112335 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112338 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112341 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112344 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112347 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112349 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112352 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112354 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112356 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112359 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112362 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112370 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112373 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112375 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.114017 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112378 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112380 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112383 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112386 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112388 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112391 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.112393 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113309 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113317 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113320 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113323 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113326 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113328 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113331 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113340 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113343 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113346 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113348 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113351 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.114466 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113354 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113357 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113360 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113362 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113365 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113368 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113370 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113373 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113376 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113378 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113381 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113383 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113386 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113388 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113392 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113396 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113400 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113403 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113407 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.114915 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113410 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113413 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113416 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113419 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113424 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113426 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113429 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113432 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113434 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113442 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113445 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113448 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113451 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113462 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113465 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113468 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113470 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113472 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113475 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.115366 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113478 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113481 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113483 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113486 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113488 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113491 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113493 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113496 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113498 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113501 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113503 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113506 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113511 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113513 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113515 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113518 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113521 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113524 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113527 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.115805 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113529 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113532 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113535 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113543 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113546 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113548 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113551 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113553 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113556 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113560 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113562 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113565 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113567 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113570 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113573 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113575 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113578 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113580 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113583 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113586 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.116337 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113588 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113590 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113593 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113595 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113598 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113600 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113603 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113606 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113612 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113614 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113617 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113619 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113622 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113624 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113627 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.113630 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114213 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114222 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114225 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114228 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.116806 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114230 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114233 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114236 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114238 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114241 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114243 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114247 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114249 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114252 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114254 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114257 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114259 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114262 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114264 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114267 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114270 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114272 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114275 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114277 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.117282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114280 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114282 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114285 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114287 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114290 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114292 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114295 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114297 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114300 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114303 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114306 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114309 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114311 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114314 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114316 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114319 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114321 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114323 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114326 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114329 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.117824 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114331 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114334 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114336 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114340 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114343 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114349 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114352 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114354 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114357 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114359 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114362 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114365 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114368 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114370 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114373 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114375 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114378 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114380 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.118330 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114383 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114385 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114388 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114390 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114393 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114399 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114401 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114404 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114406 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114411 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114414 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114416 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114419 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114422 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114425 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114429 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114431 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114434 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.118829 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114437 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114439 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114442 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114444 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114448 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114452 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114455 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114457 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114460 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114465 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114468 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114470 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114473 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114475 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114478 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114481 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114490 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114493 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.119350 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114495 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114498 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114501 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114504 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114509 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.114511 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115608 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115618 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115625 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115629 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115638 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115641 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115645 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115649 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115652 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115655 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115659 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115662 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115665 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115668 2577 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115671 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115674 2577 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115677 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:14:03.119856 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115679 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115687 2577 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115690 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115693 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115697 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115700 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115704 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115707 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115715 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115719 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115722 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115725 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115727 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115732 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115735 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115740 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115743 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115746 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115748 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115752 2577 flags.go:64] FLAG: --enable-server="true" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115755 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115762 2577 flags.go:64] FLAG: --event-burst="100" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115765 2577 flags.go:64] FLAG: --event-qps="50" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115768 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:14:03.120493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115771 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115774 2577 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115778 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115781 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115784 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115787 2577 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115790 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115793 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115796 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115799 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115802 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115804 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115807 2577 flags.go:64] FLAG: --feature-gates="" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115812 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115815 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115818 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115821 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115831 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115834 2577 flags.go:64] FLAG: --help="false" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115837 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115840 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115844 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115847 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115851 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:14:03.121205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115854 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115857 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115860 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115862 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115866 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115868 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115871 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115874 2577 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115877 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115880 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115883 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115885 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115888 2577 flags.go:64] FLAG: --lock-file="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115891 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115893 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115896 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115902 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115905 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115907 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115910 2577 flags.go:64] FLAG: --logging-format="text" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115913 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115916 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115919 2577 flags.go:64] FLAG: --manifest-url="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115922 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115926 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:14:03.121850 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115929 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115933 2577 flags.go:64] FLAG: --max-pods="110" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115936 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115939 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115942 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115945 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115948 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115952 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115954 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115962 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115965 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115967 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115970 2577 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115973 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115976 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115979 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115982 2577 flags.go:64] FLAG: --port="10250" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115985 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115987 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-095a2cc92717d065a" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115991 2577 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115994 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115997 2577 flags.go:64] FLAG: --register-node="true" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.115999 2577 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116003 2577 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116006 2577 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116009 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:14:03.122538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116012 2577 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116016 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116018 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116021 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116024 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116027 2577 flags.go:64] FLAG: --runonce="false" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116030 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116033 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116037 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116040 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116043 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116047 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116051 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116054 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116057 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116060 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116063 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116066 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116069 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116072 2577 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116075 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116080 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116083 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116094 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116099 2577 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:14:03.123237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116101 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116104 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116107 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116110 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116113 2577 flags.go:64] FLAG: --v="2" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116118 2577 flags.go:64] FLAG: --version="false" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116122 2577 flags.go:64] FLAG: --vmodule="" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116126 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116129 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116275 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116279 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116282 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116285 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116287 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116290 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116293 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116296 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116299 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116303 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116306 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116308 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.123900 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116311 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116313 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116316 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116318 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116321 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116323 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116326 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116328 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116331 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116333 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116335 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116338 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116341 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116344 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116346 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116349 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116353 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116356 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116358 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116362 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.124482 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116365 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116368 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116370 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116373 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116377 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116380 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116383 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116386 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116389 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116394 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116396 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116399 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116401 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116404 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116406 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116409 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116411 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116414 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116416 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.124967 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116419 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116421 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116424 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116426 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116429 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116431 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116434 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116436 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116439 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116442 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116445 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116447 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116453 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116455 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116458 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116461 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116463 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116466 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116468 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116471 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.125449 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116473 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116476 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116480 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116483 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116489 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116491 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116494 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116496 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116499 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116501 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116504 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116506 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116509 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116511 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116514 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116517 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116519 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116522 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.125960 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116526 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116530 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116533 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116535 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116538 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116541 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116544 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116546 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116549 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116551 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116554 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116556 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116559 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116561 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116564 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116567 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.126421 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.116595 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116630 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116634 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116637 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116640 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116643 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116646 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116649 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116651 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116654 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116657 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116660 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116665 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116667 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116670 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116672 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116675 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116678 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116680 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116683 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.126801 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116686 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116689 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116692 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116694 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116697 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116701 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116705 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116708 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116711 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116713 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116716 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116719 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116723 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116726 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116728 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116731 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116734 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116736 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116739 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.127458 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116741 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116744 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116746 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116749 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116752 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116754 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116757 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116759 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116761 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116764 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116767 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116769 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116772 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116775 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116778 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116780 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116783 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116785 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116788 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116790 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.128207 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116793 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116795 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116798 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116800 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116803 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116807 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116810 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116813 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116816 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116849 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116857 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116861 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116865 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116874 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116876 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116879 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116882 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116885 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.128792 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116888 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116892 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116898 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116901 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116904 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116907 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116910 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116912 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116915 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116918 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116920 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116923 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116926 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116928 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116931 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116937 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116940 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116943 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116946 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.129282 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116948 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116951 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116954 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116957 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116963 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116965 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116968 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116971 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116974 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.116976 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.124291 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.124305 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.124321 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.124325 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124384 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124389 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124392 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124395 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124397 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.129732 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124400 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124403 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124406 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124409 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124412 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124414 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124417 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124419 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124422 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124425 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124428 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124432 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124436 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124438 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124441 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124443 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124446 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124448 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124465 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.130225 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124469 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124472 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124475 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124478 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124480 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124483 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124489 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124492 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124495 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124497 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124500 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124502 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124504 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124507 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124509 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124512 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124514 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124517 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124519 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.130667 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124522 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124524 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124527 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124530 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124533 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124535 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124538 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124540 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124543 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124545 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124548 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124550 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124552 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124555 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124557 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124560 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124562 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124565 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124567 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124570 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.131144 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124572 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124575 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124577 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124580 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124582 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124585 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124587 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124590 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124592 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124594 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124597 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124599 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124604 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124606 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124611 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124615 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124618 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124621 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.131625 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124635 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124639 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124642 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124645 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124649 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124651 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124654 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124657 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124660 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124662 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124665 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124667 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124670 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124672 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124675 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124677 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124680 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124682 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124685 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124687 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.132099 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124690 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124692 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124694 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.124722 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124858 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124863 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124866 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124868 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124871 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124874 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124877 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124880 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124883 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124885 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124888 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124890 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.132705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124893 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124895 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124898 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124900 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124903 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124905 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124908 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124910 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124913 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124915 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124918 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124920 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124923 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124925 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124928 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124931 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124933 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124936 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124938 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.133109 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124941 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124943 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124946 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124948 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124951 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124953 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124956 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124958 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124961 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124963 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124966 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124968 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124971 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124974 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124976 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124979 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124982 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124984 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124987 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124989 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.133568 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124992 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124994 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124997 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.124999 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125001 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125004 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125007 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125009 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125011 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125014 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125016 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125019 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125022 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125024 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125027 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125029 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125031 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125034 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125036 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125038 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.134012 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125041 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125045 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125049 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125053 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125055 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125058 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125061 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125063 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125066 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125068 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125071 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125074 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125076 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125079 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125082 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125084 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125087 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125089 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125092 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.134476 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125094 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125097 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125099 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125101 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125104 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125106 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125108 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125111 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125114 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125116 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125118 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125121 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125123 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125126 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125130 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.134905 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125134 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.125173 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125206 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125210 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125214 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125216 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125220 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125223 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125226 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125228 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125234 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125237 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125240 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125243 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125246 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125249 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:03.135427 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125251 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125255 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125257 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125260 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125262 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125265 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125268 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125271 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125273 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125278 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125281 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125283 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125286 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125288 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125291 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125294 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125302 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125306 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:03.135812 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125309 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125312 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125315 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125318 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125321 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125323 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125326 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125330 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125333 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125336 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125338 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125341 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125344 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125346 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125349 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125352 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125355 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125358 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125360 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:03.136273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125363 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125365 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125368 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125370 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125373 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125375 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125378 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125381 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125383 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125386 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125389 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125393 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125397 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125400 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125403 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125406 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125409 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125411 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125414 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:03.136705 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125416 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125420 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125423 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125425 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125428 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125430 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125432 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125435 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125437 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125440 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125444 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125446 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125449 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125452 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125454 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125457 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125459 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125462 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125465 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125467 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:03.137169 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125470 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125472 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125475 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125477 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125480 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125482 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125485 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125489 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125491 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125494 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125496 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125499 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125501 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125504 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:03.125508 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.125555 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.129685 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.130761 2577 server.go:1014] "Starting client certificate rotation" Oct 29 22:14:03.137636 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.130849 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:03.138047 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.130882 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:03.138047 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.132919 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:03.138047 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.134055 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:03.144965 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.144948 2577 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:14:03.148718 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.148704 2577 log.go:25] "Validated CRI v1 image API" Oct 29 22:14:03.150481 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.150460 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:14:03.153224 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.153207 2577 fs.go:135] Filesystem UUIDs: map[2d7c4d52-a99d-4ad1-b48d-e66e5a40fb59:/dev/nvme0n1p3 454241e5-f75e-4fbd-aadc-475895045f39:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 29 22:14:03.153270 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.153224 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:14:03.159909 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.159800 2577 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:03.158345358 +0000 UTC m=+0.371696052 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101612 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2537848478bde51310c38b6cece4aa SystemUUID:ec253784-8478-bde5-1310-c38b6cece4aa BootID:6e09bfe6-7c0e-464d-b358-fdcb2e66c23e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:1a:85:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:1a:85:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:e8:09:49:8d:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:14:03.160592 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.160581 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:14:03.160673 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.160661 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:14:03.164127 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.164102 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:14:03.164269 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.164130 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:14:03.165093 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.165084 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:14:03.165123 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.165096 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:14:03.165123 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.165107 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:03.165123 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.165117 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:03.166576 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.166566 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:14:03.171196 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.171184 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:14:03.171320 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.171310 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:14:03.172177 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.172167 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:14:03.172229 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.172183 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:14:03.172229 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.172203 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:14:03.172229 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.172215 2577 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:14:03.172229 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.172224 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:14:03.172551 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.172540 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:03.175046 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.175025 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:14:03.183327 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.183308 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:03.184293 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184279 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:14:03.184344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184312 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:14:03.184474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184463 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:14:03.184516 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184479 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:14:03.184516 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184485 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:14:03.184516 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184491 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:14:03.184516 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184496 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:14:03.184516 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.184501 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:14:03.185184 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.185175 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:14:03.185213 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.185185 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:14:03.185213 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.185192 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:14:03.185213 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.185199 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:14:03.185213 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.185207 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:14:03.185376 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.185215 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:14:03.186093 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.186082 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:14:03.186124 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.186096 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:14:03.189980 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.189965 2577 server.go:1263] "Started kubelet" Oct 29 22:14:03.190164 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.190070 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:14:03.190205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.190110 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:14:03.190424 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.190189 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:14:03.190737 ip-10-0-134-94 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:14:03.191846 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.191831 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:14:03.193790 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.193775 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:14:03.196717 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.196691 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:14:03.199196 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.199177 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:03.200539 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.200523 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:14:03.200653 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.200628 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.201309 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201290 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:14:03.201309 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201291 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:14:03.201435 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201320 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:14:03.201435 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201427 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:14:03.201435 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201434 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:14:03.201527 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201487 2577 factory.go:153] Registering CRI-O factory Oct 29 22:14:03.201527 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201499 2577 factory.go:223] Registration of the crio container factory successfully Oct 29 22:14:03.201578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201544 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:14:03.201578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201551 2577 factory.go:55] Registering systemd factory Oct 29 22:14:03.201578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201556 2577 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:14:03.201578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201569 2577 factory.go:103] Registering Raw factory Oct 29 22:14:03.201578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.201579 2577 manager.go:1199] Started watching for new ooms in manager Oct 29 22:14:03.202703 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.202676 2577 manager.go:319] Starting recovery of all containers Oct 29 22:14:03.217000 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.216984 2577 manager.go:324] Recovery completed Oct 29 22:14:03.220879 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.220865 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.222783 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.222755 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.222865 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.222799 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.222865 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.222813 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.223485 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.223468 2577 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:14:03.223485 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.223482 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:14:03.223576 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.223499 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:14:03.224225 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.224197 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:14:03.224225 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.224203 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:14:03.224379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.224244 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.224975 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.224959 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.225041 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.224987 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.225041 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.224997 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.225314 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.224196 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-94.ec2.internal.18e31fcd49c41812 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-94.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-94.ec2.internal,},FirstTimestamp:2026-10-29 22:14:03.189934098 +0000 UTC m=+0.403284791,LastTimestamp:2026-10-29 22:14:03.189934098 +0000 UTC m=+0.403284791,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-94.ec2.internal,}" Oct 29 22:14:03.225591 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.225580 2577 policy_none.go:50] "Start" Oct 29 22:14:03.225620 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.225596 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:14:03.225620 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.225606 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:14:03.227583 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.227573 2577 policy_none.go:44] "Start" Oct 29 22:14:03.228418 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.228399 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:14:03.242738 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.242576 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vttp7" Oct 29 22:14:03.246895 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.246868 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 22:14:03.246987 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.246834 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-94.ec2.internal.18e31fcd4bb94ff4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-94.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-94.ec2.internal,},FirstTimestamp:2026-10-29 22:14:03.22278194 +0000 UTC m=+0.436132640,LastTimestamp:2026-10-29 22:14:03.22278194 +0000 UTC m=+0.436132640,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-94.ec2.internal,}" Oct 29 22:14:03.247091 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.247068 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 22:14:03.247176 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.247128 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:14:03.249348 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.249331 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vttp7" Oct 29 22:14:03.263720 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.263691 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:14:03.266264 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.266251 2577 manager.go:341] "Starting Device Plugin manager" Oct 29 22:14:03.266977 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.266963 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:14:03.266977 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.266978 2577 server.go:87] "Starting device plugin registration server" Oct 29 22:14:03.267199 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.267189 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:14:03.267232 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.267200 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:14:03.267311 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.267298 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:14:03.267386 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.267371 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:14:03.267386 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.267382 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:14:03.267889 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.267869 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:14:03.267957 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.267914 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.341695 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.341666 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:14:03.343668 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.342992 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:14:03.343668 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.343010 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:14:03.343668 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.343028 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:14:03.343668 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.343064 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 22:14:03.355100 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.355085 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:03.367441 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.367402 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.368297 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.368279 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.368401 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.368313 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.368401 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.368328 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.368401 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.368360 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.385135 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.385119 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.385253 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.385144 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-94.ec2.internal\": node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.419654 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.419630 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.443749 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.443727 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal"] Oct 29 22:14:03.443853 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.443792 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.445015 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.444999 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.445108 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.445033 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.445108 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.445047 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.447441 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.447427 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.447588 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.447573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.447634 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.447609 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.448344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.448329 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.448425 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.448347 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.448425 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.448356 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.448425 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.448371 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.448425 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.448374 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.448425 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.448385 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.450540 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.450524 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.450610 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.450549 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:03.451551 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.451536 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:03.451653 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.451565 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:03.451653 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.451578 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:03.486559 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.486535 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-94.ec2.internal\" not found" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.488534 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.488517 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-94.ec2.internal\" not found" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.520735 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.520713 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.602237 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.602211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.602384 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.602247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.602384 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.602271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6ac7ba86bb940ac627127f0b9857b88-config\") pod \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" (UID: \"d6ac7ba86bb940ac627127f0b9857b88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.621388 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.621334 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.703058 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.703019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.703058 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.703061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.703300 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.703079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6ac7ba86bb940ac627127f0b9857b88-config\") pod \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" (UID: \"d6ac7ba86bb940ac627127f0b9857b88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.703300 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.703184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.703300 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.703214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6ac7ba86bb940ac627127f0b9857b88-config\") pod \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" (UID: \"d6ac7ba86bb940ac627127f0b9857b88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.703300 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.703194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.722165 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.722121 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.790315 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.790296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.791409 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:03.791371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:03.822430 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.822403 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:03.922936 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:03.922851 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:04.023411 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.023378 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:04.087194 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.087171 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:04.101392 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.101365 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:04.107421 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.107400 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:04.122820 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.122803 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:04.123532 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.123518 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:04.130867 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.130840 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:14:04.130986 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.130966 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:04.131050 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.131003 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:04.131050 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.131013 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:04.131111 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.131051 2577 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a5b593b5fe01a4b459dee98cc65e601a-e829f27606018762.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.134.94:43840->3.148.110.236:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:04.173116 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.173067 2577 apiserver.go:52] "Watching apiserver" Oct 29 22:14:04.199474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.199450 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:04.243120 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.243097 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:14:04.243462 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.243445 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal","openshift-multus/multus-7qnrm","openshift-multus/network-metrics-daemon-8j2s2","openshift-cluster-node-tuning-operator/tuned-5jrfz","openshift-dns/node-resolver-tbx55","openshift-multus/multus-additional-cni-plugins-t7vcg","openshift-network-operator/iptables-alerter-lc8wg","openshift-ovn-kubernetes/ovnkube-node-hx452","kube-system/konnectivity-agent-mmmgd","openshift-image-registry/node-ca-2cvqq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal","openshift-network-diagnostics/network-check-target-k2gxs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d","kube-system/global-pull-secret-syncer-px4td"] Oct 29 22:14:04.245830 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.245811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.247937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.247922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.250692 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.250670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.250766 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.250722 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:04.253598 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.253583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.255673 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.255634 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:09:03 +0000 UTC" deadline="2028-05-27 13:22:00.152008846 +0000 UTC" Oct 29 22:14:04.255736 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.255674 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13815h7m55.896337634s" Oct 29 22:14:04.255979 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.255965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.258226 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.258204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.259886 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.259866 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:14:04.259996 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.259879 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.259996 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.259899 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbljw\"" Oct 29 22:14:04.259996 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.259915 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:14:04.259996 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.259929 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.260173 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.260006 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:14:04.260482 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.260465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.261349 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.261175 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:14:04.262678 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.262660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.264889 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.264875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.267457 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.267438 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2cw6s\"" Oct 29 22:14:04.267995 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.267972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:04.268074 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.268041 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:04.270538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.270515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.270793 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.270737 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:14:04.270793 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.270787 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dj4ww" Oct 29 22:14:04.272946 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.272929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.273032 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.272993 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:04.289953 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.289921 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ac7ba86bb940ac627127f0b9857b88.slice/crio-fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1 WatchSource:0}: Error finding container fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1: Status 404 returned error can't find the container with id fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1 Oct 29 22:14:04.290326 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.290305 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:04.290523 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.290504 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaccf67d50dfb71546a9fa0941d51e11.slice/crio-3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187 WatchSource:0}: Error finding container 3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187: Status 404 returned error can't find the container with id 3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187 Oct 29 22:14:04.295290 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.295275 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:14:04.302723 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.302703 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:14:04.305584 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-hostroot\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.305677 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-slash\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.305677 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbeceea4-8081-4c42-a085-07b5457ed5c7-host\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.305768 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.305768 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40badafa-2bba-47cc-b94c-c8f7233c7fb6-agent-certs\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.305859 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtmn\" (UniqueName: \"kubernetes.io/projected/0762bfee-d0ac-4a88-bee5-212f3b230a2d-kube-api-access-vqtmn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.305909 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.305957 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.306022 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.305978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbdk\" (UniqueName: \"kubernetes.io/projected/7c439b93-fb4b-441d-96d5-54852b26dcde-kube-api-access-dtbdk\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.306139 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bdb67-9b1d-40f0-8d58-db1157201e45-tmp-dir\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.306139 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:04.306139 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-log-socket\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.306139 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-k8s-cni-cncf-io\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6vw\" (UniqueName: \"kubernetes.io/projected/49aa972a-477a-419c-a2ba-9e5195039423-kube-api-access-hx6vw\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79nh\" (UniqueName: \"kubernetes.io/projected/d5414f15-670a-4ca7-a6c2-78a5426d717e-kube-api-access-m79nh\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-host-slash\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-socket-dir-parent\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-sys-fs\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-conf\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.306366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-ovn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-netd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-script-lib\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-etc-selinux\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-system-cni-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-system-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-registration-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-kubelet-config\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-sys\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.306740 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-cnibin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-systemd-units\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-netns\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45sq\" (UniqueName: \"kubernetes.io/projected/fbeceea4-8081-4c42-a085-07b5457ed5c7-kube-api-access-b45sq\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-modprobe-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-config\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-host\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.306973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-conf-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-node-log\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-socket-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-cnibin\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwk7k\" (UniqueName: \"kubernetes.io/projected/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-kube-api-access-xwk7k\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.307214 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwrp\" (UniqueName: \"kubernetes.io/projected/340bdb67-9b1d-40f0-8d58-db1157201e45-kube-api-access-zkwrp\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-kubelet\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-env-overrides\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-systemd\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwdzn\" (UniqueName: \"kubernetes.io/projected/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-kube-api-access-fwdzn\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-multus\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307361 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-systemd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-bin\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbeceea4-8081-4c42-a085-07b5457ed5c7-serviceca\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-kubelet\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-device-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-tmp\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-multus-certs\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-dbus\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysconfig\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.307681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-run\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340bdb67-9b1d-40f0-8d58-db1157201e45-hosts-file\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-netns\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-etc-kubernetes\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-etc-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-bin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-lib-modules\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-iptables-alerter-script\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-daemon-config\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.307971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29lf\" (UniqueName: \"kubernetes.io/projected/6813a426-c6a9-4a87-b2b4-8d60472da95d-kube-api-access-n29lf\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-kubernetes\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-var-lib-kubelet\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-cni-binary-copy\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-var-lib-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.308171 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-tuned\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.308730 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.308184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40badafa-2bba-47cc-b94c-c8f7233c7fb6-konnectivity-ca\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.346198 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.346137 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" event={"ID":"d6ac7ba86bb940ac627127f0b9857b88","Type":"ContainerStarted","Data":"fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1"} Oct 29 22:14:04.347212 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.347192 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" event={"ID":"daccf67d50dfb71546a9fa0941d51e11","Type":"ContainerStarted","Data":"3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187"} Oct 29 22:14:04.360118 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360101 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:14:04.360428 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360403 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:14:04.360516 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360507 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kk6hd\"" Oct 29 22:14:04.360568 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360529 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hznbv\"" Oct 29 22:14:04.360834 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360808 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:14:04.360954 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360930 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:14:04.361014 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360936 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.361014 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360988 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkqw\"" Oct 29 22:14:04.361014 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.360999 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.361163 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361048 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:14:04.361163 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361061 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxx6w\"" Oct 29 22:14:04.361163 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361070 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.361163 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361060 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.361163 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361077 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.361163 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361124 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w24c4\"" Oct 29 22:14:04.361452 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361423 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:14:04.361622 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361600 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.361694 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361607 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.361785 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361767 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.361847 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.361820 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm48d\"" Oct 29 22:14:04.362083 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362043 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.362083 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362062 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:14:04.362310 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362048 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.362310 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362263 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:14:04.362310 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362175 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-475gp\"" Oct 29 22:14:04.362482 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362467 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:14:04.362543 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.362527 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:04.372886 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.372870 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:14:04.409100 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409182 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-config\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409182 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-host\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.409182 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-conf-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409182 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-node-log\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-socket-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-cnibin\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwk7k\" (UniqueName: \"kubernetes.io/projected/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-kube-api-access-xwk7k\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409230 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409243 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-conf-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409264 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-node-log\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409287 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-cnibin\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwrp\" (UniqueName: \"kubernetes.io/projected/340bdb67-9b1d-40f0-8d58-db1157201e45-kube-api-access-zkwrp\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-host\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-kubelet\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-env-overrides\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409354 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-socket-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.409379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-systemd\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409391 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-kubelet\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdzn\" (UniqueName: \"kubernetes.io/projected/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-kube-api-access-fwdzn\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-multus\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-systemd\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-systemd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-bin\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbeceea4-8081-4c42-a085-07b5457ed5c7-serviceca\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409489 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-multus\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-systemd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-bin\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-kubelet\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-device-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409614 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-tmp\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409621 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-kubelet\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-multus-certs\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-multus-certs\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-device-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.409955 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-dbus\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409741 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-env-overrides\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysconfig\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-config\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-run\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340bdb67-9b1d-40f0-8d58-db1157201e45-hosts-file\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysconfig\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-netns\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-etc-kubernetes\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340bdb67-9b1d-40f0-8d58-db1157201e45-hosts-file\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-etc-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-bin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409912 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-netns\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409918 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-run\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409941 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-dbus\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409953 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-etc-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.410719 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-lib-modules\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-iptables-alerter-script\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409993 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbeceea4-8081-4c42-a085-07b5457ed5c7-serviceca\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409978 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-daemon-config\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410017 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.409996 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-etc-kubernetes\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-lib-modules\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410061 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-bin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29lf\" (UniqueName: \"kubernetes.io/projected/6813a426-c6a9-4a87-b2b4-8d60472da95d-kube-api-access-n29lf\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-kubernetes\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-var-lib-kubelet\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-cni-binary-copy\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-var-lib-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-kubernetes\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-tuned\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40badafa-2bba-47cc-b94c-c8f7233c7fb6-konnectivity-ca\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.411611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-hostroot\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-slash\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbeceea4-8081-4c42-a085-07b5457ed5c7-host\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40badafa-2bba-47cc-b94c-c8f7233c7fb6-agent-certs\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtmn\" (UniqueName: \"kubernetes.io/projected/0762bfee-d0ac-4a88-bee5-212f3b230a2d-kube-api-access-vqtmn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbdk\" (UniqueName: \"kubernetes.io/projected/7c439b93-fb4b-441d-96d5-54852b26dcde-kube-api-access-dtbdk\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-var-lib-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bdb67-9b1d-40f0-8d58-db1157201e45-tmp-dir\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-log-socket\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-k8s-cni-cncf-io\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410657 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-var-lib-kubelet\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6vw\" (UniqueName: \"kubernetes.io/projected/49aa972a-477a-419c-a2ba-9e5195039423-kube-api-access-hx6vw\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.412446 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79nh\" (UniqueName: \"kubernetes.io/projected/d5414f15-670a-4ca7-a6c2-78a5426d717e-kube-api-access-m79nh\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-host-slash\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-socket-dir-parent\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-sys-fs\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-conf\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-ovn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-netd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-script-lib\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-etc-selinux\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-system-cni-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-system-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-registration-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-kubelet-config\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.413223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410986 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-slash\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.410999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbeceea4-8081-4c42-a085-07b5457ed5c7-host\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-sys\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-cnibin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411075 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-log-socket\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-systemd-units\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411141 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-netd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-netns\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411219 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45sq\" (UniqueName: \"kubernetes.io/projected/fbeceea4-8081-4c42-a085-07b5457ed5c7-kube-api-access-b45sq\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-systemd-units\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411249 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-daemon-config\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411257 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-k8s-cni-cncf-io\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-netns\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411341 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-ovn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411349 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-sys-fs\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.413931 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.411415 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.411482 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret podName:5f6c3496-906d-4d23-8e0c-a518b01f9ca3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:04.911463938 +0000 UTC m=+2.124814639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret") pod "global-pull-secret-syncer-px4td" (UID: "5f6c3496-906d-4d23-8e0c-a518b01f9ca3") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-modprobe-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411526 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-system-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411599 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-conf\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-modprobe-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-host-slash\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.411664 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411606 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-kubelet-config\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411682 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-socket-dir-parent\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411703 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-registration-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411731 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-sys\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.411735 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:04.911719606 +0000 UTC m=+2.125070295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411763 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-etc-selinux\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.414451 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-hostroot\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411823 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-system-cni-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.411841 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-cnibin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412241 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bdb67-9b1d-40f0-8d58-db1157201e45-tmp-dir\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412369 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-cni-binary-copy\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412398 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40badafa-2bba-47cc-b94c-c8f7233c7fb6-konnectivity-ca\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-script-lib\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.412976 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-iptables-alerter-script\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.413136 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.413947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-tmp\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.414196 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-tuned\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.414463 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.414937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.414874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40badafa-2bba-47cc-b94c-c8f7233c7fb6-agent-certs\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.428623 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.428565 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:04.428623 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.428585 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:04.428623 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.428596 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xct6r for pod openshift-network-diagnostics/network-check-target-k2gxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:04.428835 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.428660 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r podName:93780727-5cdc-4eaa-91fc-92e7c3889eaa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:04.928645151 +0000 UTC m=+2.141995853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xct6r" (UniqueName: "kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r") pod "network-check-target-k2gxs" (UID: "93780727-5cdc-4eaa-91fc-92e7c3889eaa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:04.428898 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.428886 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwk7k\" (UniqueName: \"kubernetes.io/projected/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-kube-api-access-xwk7k\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.429730 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.429710 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbdk\" (UniqueName: \"kubernetes.io/projected/7c439b93-fb4b-441d-96d5-54852b26dcde-kube-api-access-dtbdk\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.429814 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.429727 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdzn\" (UniqueName: \"kubernetes.io/projected/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-kube-api-access-fwdzn\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.429814 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.429707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29lf\" (UniqueName: \"kubernetes.io/projected/6813a426-c6a9-4a87-b2b4-8d60472da95d-kube-api-access-n29lf\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.429814 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.429746 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwrp\" (UniqueName: \"kubernetes.io/projected/340bdb67-9b1d-40f0-8d58-db1157201e45-kube-api-access-zkwrp\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.431279 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.431261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6vw\" (UniqueName: \"kubernetes.io/projected/49aa972a-477a-419c-a2ba-9e5195039423-kube-api-access-hx6vw\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.431597 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.431579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtmn\" (UniqueName: \"kubernetes.io/projected/0762bfee-d0ac-4a88-bee5-212f3b230a2d-kube-api-access-vqtmn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.432031 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.432009 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45sq\" (UniqueName: \"kubernetes.io/projected/fbeceea4-8081-4c42-a085-07b5457ed5c7-kube-api-access-b45sq\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.432107 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.432027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79nh\" (UniqueName: \"kubernetes.io/projected/d5414f15-670a-4ca7-a6c2-78a5426d717e-kube-api-access-m79nh\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.581469 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.581439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:04.589761 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.589734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qnrm" Oct 29 22:14:04.592120 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.592097 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c439b93_fb4b_441d_96d5_54852b26dcde.slice/crio-c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec WatchSource:0}: Error finding container c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec: Status 404 returned error can't find the container with id c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec Oct 29 22:14:04.598530 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.598509 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6813a426_c6a9_4a87_b2b4_8d60472da95d.slice/crio-1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2 WatchSource:0}: Error finding container 1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2: Status 404 returned error can't find the container with id 1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2 Oct 29 22:14:04.604175 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.604152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:04.609218 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.609203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:04.615312 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.615287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:04.616208 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.616183 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8709f8_a9fb_4dbe_90ea_2be488f64dc0.slice/crio-7ab1f57bba17a43c38763e95f8855919da1bb00fe534c2c543c5a9bad4d94920 WatchSource:0}: Error finding container 7ab1f57bba17a43c38763e95f8855919da1bb00fe534c2c543c5a9bad4d94920: Status 404 returned error can't find the container with id 7ab1f57bba17a43c38763e95f8855919da1bb00fe534c2c543c5a9bad4d94920 Oct 29 22:14:04.620442 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.620420 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda625d89c_ea34_4bfb_80d6_3bcbf5958f5c.slice/crio-ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72 WatchSource:0}: Error finding container ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72: Status 404 returned error can't find the container with id ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72 Oct 29 22:14:04.625781 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.625755 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340bdb67_9b1d_40f0_8d58_db1157201e45.slice/crio-4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52 WatchSource:0}: Error finding container 4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52: Status 404 returned error can't find the container with id 4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52 Oct 29 22:14:04.630456 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.630416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:04.632626 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.632604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:04.639071 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.639048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:04.644576 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.644554 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:04.644758 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.644734 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40badafa_2bba_47cc_b94c_c8f7233c7fb6.slice/crio-b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c WatchSource:0}: Error finding container b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c: Status 404 returned error can't find the container with id b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c Oct 29 22:14:04.645778 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.645706 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeceea4_8081_4c42_a085_07b5457ed5c7.slice/crio-417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92 WatchSource:0}: Error finding container 417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92: Status 404 returned error can't find the container with id 417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92 Oct 29 22:14:04.654130 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.654102 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0762bfee_d0ac_4a88_bee5_212f3b230a2d.slice/crio-13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab WatchSource:0}: Error finding container 13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab: Status 404 returned error can't find the container with id 13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab Oct 29 22:14:04.658377 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:04.658333 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49aa972a_477a_419c_a2ba_9e5195039423.slice/crio-f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c WatchSource:0}: Error finding container f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c: Status 404 returned error can't find the container with id f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c Oct 29 22:14:04.916718 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.916669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:04.916899 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:04.916774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:04.917040 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.916915 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:04.917040 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.916979 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret podName:5f6c3496-906d-4d23-8e0c-a518b01f9ca3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:05.916959936 +0000 UTC m=+3.130310633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret") pod "global-pull-secret-syncer-px4td" (UID: "5f6c3496-906d-4d23-8e0c-a518b01f9ca3") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:04.917446 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.917428 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:04.917548 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:04.917489 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:05.917471711 +0000 UTC m=+3.130822411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:05.020308 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.019594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:05.020308 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.019829 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:05.020308 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.019851 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:05.020308 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.019865 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xct6r for pod openshift-network-diagnostics/network-check-target-k2gxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:05.020308 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.019929 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r podName:93780727-5cdc-4eaa-91fc-92e7c3889eaa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:06.019907937 +0000 UTC m=+3.233258621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xct6r" (UniqueName: "kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r") pod "network-check-target-k2gxs" (UID: "93780727-5cdc-4eaa-91fc-92e7c3889eaa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:05.274453 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.274322 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:04 +0000 UTC" deadline="2028-07-08 07:40:14.340927936 +0000 UTC" Oct 29 22:14:05.274453 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.274394 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14817h26m9.066537832s" Oct 29 22:14:05.359883 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.359839 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" event={"ID":"49aa972a-477a-419c-a2ba-9e5195039423","Type":"ContainerStarted","Data":"f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c"} Oct 29 22:14:05.368161 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.368125 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lc8wg" event={"ID":"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c","Type":"ContainerStarted","Data":"ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72"} Oct 29 22:14:05.376284 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.376189 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" event={"ID":"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0","Type":"ContainerStarted","Data":"7ab1f57bba17a43c38763e95f8855919da1bb00fe534c2c543c5a9bad4d94920"} Oct 29 22:14:05.393581 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.393466 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab"} Oct 29 22:14:05.405489 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.405456 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cvqq" event={"ID":"fbeceea4-8081-4c42-a085-07b5457ed5c7","Type":"ContainerStarted","Data":"417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92"} Oct 29 22:14:05.435046 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.435005 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mmmgd" event={"ID":"40badafa-2bba-47cc-b94c-c8f7233c7fb6","Type":"ContainerStarted","Data":"b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c"} Oct 29 22:14:05.454133 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.454096 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tbx55" event={"ID":"340bdb67-9b1d-40f0-8d58-db1157201e45","Type":"ContainerStarted","Data":"4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52"} Oct 29 22:14:05.463809 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.463764 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qnrm" event={"ID":"6813a426-c6a9-4a87-b2b4-8d60472da95d","Type":"ContainerStarted","Data":"1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2"} Oct 29 22:14:05.478455 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.478417 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerStarted","Data":"c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec"} Oct 29 22:14:05.674540 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.674449 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:05.929797 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.929709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:05.929797 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:05.929776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:05.930029 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.929918 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:05.930029 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.929987 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:07.929968163 +0000 UTC m=+5.143318854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:05.930454 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.930433 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:05.930564 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:05.930503 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret podName:5f6c3496-906d-4d23-8e0c-a518b01f9ca3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:07.930485795 +0000 UTC m=+5.143836493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret") pod "global-pull-secret-syncer-px4td" (UID: "5f6c3496-906d-4d23-8e0c-a518b01f9ca3") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:06.002283 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.002245 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:06.031021 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.030988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:06.031251 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.031224 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:06.031251 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.031247 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:06.031383 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.031261 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xct6r for pod openshift-network-diagnostics/network-check-target-k2gxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:06.031383 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.031324 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r podName:93780727-5cdc-4eaa-91fc-92e7c3889eaa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.031304067 +0000 UTC m=+5.244654774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xct6r" (UniqueName: "kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r") pod "network-check-target-k2gxs" (UID: "93780727-5cdc-4eaa-91fc-92e7c3889eaa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:06.274757 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.274719 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:04 +0000 UTC" deadline="2028-08-13 18:24:52.984942474 +0000 UTC" Oct 29 22:14:06.274757 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.274752 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15692h10m46.710193593s" Oct 29 22:14:06.343619 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.343587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:06.343808 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.343723 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:06.344249 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.344209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:06.344383 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.344361 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:06.344511 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:06.344438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:06.344623 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:06.344515 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:07.275684 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:07.275649 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:07.951263 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:07.951103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:07.951263 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:07.951228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:07.951455 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:07.951382 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:07.951455 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:07.951450 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret podName:5f6c3496-906d-4d23-8e0c-a518b01f9ca3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.951431894 +0000 UTC m=+9.164782600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret") pod "global-pull-secret-syncer-px4td" (UID: "5f6c3496-906d-4d23-8e0c-a518b01f9ca3") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:07.951898 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:07.951879 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:07.951972 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:07.951940 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.951921031 +0000 UTC m=+9.165271719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:08.052116 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:08.052044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:08.052300 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.052227 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:08.052300 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.052247 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:08.052300 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.052257 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xct6r for pod openshift-network-diagnostics/network-check-target-k2gxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:08.052476 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.052308 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r podName:93780727-5cdc-4eaa-91fc-92e7c3889eaa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:12.052294688 +0000 UTC m=+9.265645387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xct6r" (UniqueName: "kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r") pod "network-check-target-k2gxs" (UID: "93780727-5cdc-4eaa-91fc-92e7c3889eaa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:08.343933 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:08.343851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:08.344460 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.343978 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:08.344460 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:08.344380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:08.344583 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.344482 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:08.344583 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:08.344535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:08.344694 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:08.344635 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:09.511992 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.511781 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" event={"ID":"d6ac7ba86bb940ac627127f0b9857b88","Type":"ContainerStarted","Data":"26846ecbd2ad1f1e773d40d4f79a94adab765e12584e2b2f24d482a6294fef70"} Oct 29 22:14:09.515797 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.515762 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mmmgd" event={"ID":"40badafa-2bba-47cc-b94c-c8f7233c7fb6","Type":"ContainerStarted","Data":"22bd7ea82a7894426fdc21a30d2e814ed39d771154eb4d906fe01864a5d9f69e"} Oct 29 22:14:09.518872 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.518843 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="1aa8b72d320aa24b7abc7e2c3b8fd7ebd9b1d35de25b3c26a6efa8381b68c91b" exitCode=0 Oct 29 22:14:09.518998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.518919 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerDied","Data":"1aa8b72d320aa24b7abc7e2c3b8fd7ebd9b1d35de25b3c26a6efa8381b68c91b"} Oct 29 22:14:09.522092 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.522065 2577 generic.go:356] "Generic (PLEG): container finished" podID="daccf67d50dfb71546a9fa0941d51e11" containerID="88b68f8819125574d4bb62968276b1bfcf0a9e5ebf383c4df331f22fc8cfa085" exitCode=0 Oct 29 22:14:09.522178 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.522106 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" event={"ID":"daccf67d50dfb71546a9fa0941d51e11","Type":"ContainerDied","Data":"88b68f8819125574d4bb62968276b1bfcf0a9e5ebf383c4df331f22fc8cfa085"} Oct 29 22:14:09.559182 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:09.559098 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" podStartSLOduration=5.55907891 podStartE2EDuration="5.55907891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:09.557872096 +0000 UTC m=+6.771222804" watchObservedRunningTime="2026-10-29 22:14:09.55907891 +0000 UTC m=+6.772429619" Oct 29 22:14:10.344300 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.344269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:10.344442 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:10.344406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:10.344827 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.344622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:10.344827 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:10.344728 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:10.345004 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.344861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:10.345004 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:10.344974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:10.535575 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.535533 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" event={"ID":"daccf67d50dfb71546a9fa0941d51e11","Type":"ContainerStarted","Data":"aed67bf1a7b2a60f11f7d2ef9b644806206c2c33c4b80e0a120a23774a82c0e5"} Oct 29 22:14:10.546286 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.546198 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cvqq" event={"ID":"fbeceea4-8081-4c42-a085-07b5457ed5c7","Type":"ContainerStarted","Data":"006a680f09a4b14f653bb8c6d70795c78839a8f74604ec397cadab1d5f3157cb"} Oct 29 22:14:10.575550 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.575489 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mmmgd" podStartSLOduration=2.078126038 podStartE2EDuration="6.575470628s" totalImagesPullingTime="4.49734459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:09.683999941 +0000 UTC m=+6.897350650" watchObservedRunningTime="2026-10-29 22:14:10.575470628 +0000 UTC m=+7.788821338" Oct 29 22:14:10.614551 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:10.614452 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" podStartSLOduration=6.614432268 podStartE2EDuration="6.614432268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:10.576163225 +0000 UTC m=+7.789513932" watchObservedRunningTime="2026-10-29 22:14:10.614432268 +0000 UTC m=+7.827782978" Oct 29 22:14:11.553216 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:11.552416 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" event={"ID":"49aa972a-477a-419c-a2ba-9e5195039423","Type":"ContainerStarted","Data":"9327792394efdef285027e8de6af67ed11f318e053f78f09d481a41a3698c444"} Oct 29 22:14:11.956716 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:11.956581 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:11.957677 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:11.957654 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:11.984251 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:11.984203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:11.984391 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:11.984269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:11.984863 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:11.984838 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:11.984962 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:11.984907 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret podName:5f6c3496-906d-4d23-8e0c-a518b01f9ca3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.984888643 +0000 UTC m=+17.198239337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret") pod "global-pull-secret-syncer-px4td" (UID: "5f6c3496-906d-4d23-8e0c-a518b01f9ca3") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:11.985025 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:11.984969 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:11.985025 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:11.985001 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.984991229 +0000 UTC m=+17.198341924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:11.996501 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:11.996444 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2cvqq" podStartSLOduration=2.452379683 podStartE2EDuration="7.996426217s" totalImagesPullingTime="5.544046534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:10.615801931 +0000 UTC m=+7.829152638" watchObservedRunningTime="2026-10-29 22:14:11.996426217 +0000 UTC m=+9.209776921" Oct 29 22:14:12.085448 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:12.085403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:12.085642 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.085609 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:12.085642 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.085630 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:12.085642 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.085643 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xct6r for pod openshift-network-diagnostics/network-check-target-k2gxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:12.085794 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.085697 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r podName:93780727-5cdc-4eaa-91fc-92e7c3889eaa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.085678472 +0000 UTC m=+17.299029170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xct6r" (UniqueName: "kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r") pod "network-check-target-k2gxs" (UID: "93780727-5cdc-4eaa-91fc-92e7c3889eaa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:12.343634 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:12.343445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:12.343634 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.343575 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:12.344204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:12.343959 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:12.344204 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.344070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:12.344204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:12.344126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:12.344423 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:12.344211 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:12.811965 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:12.811935 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:14:13.272707 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.272032 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:12.811959966Z","UUID":"c8d562e5-dcfe-426d-bb6f-42875ef097e0","Handler":null,"Name":"","Endpoint":""} Oct 29 22:14:13.274140 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.273756 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:14:13.274140 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.273784 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:14:13.564977 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.564938 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" event={"ID":"49aa972a-477a-419c-a2ba-9e5195039423","Type":"ContainerStarted","Data":"355297c20d580d4c04976b4b6d39d29444d2db9850201e7c414b500e45726290"} Oct 29 22:14:13.567544 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.567517 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tbx55" event={"ID":"340bdb67-9b1d-40f0-8d58-db1157201e45","Type":"ContainerStarted","Data":"749c77e8e967b4383ea2b6db6c34983391367281a79d7281342e18d6b3474323"} Oct 29 22:14:13.570590 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.570568 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qnrm" event={"ID":"6813a426-c6a9-4a87-b2b4-8d60472da95d","Type":"ContainerStarted","Data":"8ccc7ff3eaf632f4f8f824b65315b6495036bc2fd24d3ed83fe08be6d34e9074"} Oct 29 22:14:13.650522 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.650464 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7qnrm" podStartSLOduration=2.104121281 podStartE2EDuration="10.650446013s" totalImagesPullingTime="8.546324732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.650060312 +0000 UTC m=+10.863411019" watchObservedRunningTime="2026-10-29 22:14:13.650446013 +0000 UTC m=+10.863796717" Oct 29 22:14:13.650799 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:13.650760 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tbx55" podStartSLOduration=1.202958195 podStartE2EDuration="9.650747771s" totalImagesPullingTime="8.447789576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.612412721 +0000 UTC m=+10.825763423" watchObservedRunningTime="2026-10-29 22:14:13.650747771 +0000 UTC m=+10.864098474" Oct 29 22:14:14.343820 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.343548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:14.343820 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.343798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:14.344267 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.343754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:14.344267 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:14.343895 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:14.344267 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:14.344015 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:14.344267 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:14.344095 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:14.575692 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.575597 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" event={"ID":"49aa972a-477a-419c-a2ba-9e5195039423","Type":"ContainerStarted","Data":"0b53780c8f524f31fb09381eaf67035851d31bf0c92e73b5af0c6d278cec44ac"} Oct 29 22:14:14.578002 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.577966 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="98529ed38296839528aaa4c7f492896b6a3ebadaae855e51f62e6f2cc6c15d58" exitCode=0 Oct 29 22:14:14.578140 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.578046 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerDied","Data":"98529ed38296839528aaa4c7f492896b6a3ebadaae855e51f62e6f2cc6c15d58"} Oct 29 22:14:14.615412 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:14.615306 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" podStartSLOduration=1.712885423 podStartE2EDuration="10.615288635s" totalImagesPullingTime="8.902403212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:14.614593838 +0000 UTC m=+11.827944542" watchObservedRunningTime="2026-10-29 22:14:14.615288635 +0000 UTC m=+11.828639339" Oct 29 22:14:15.581545 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:15.581508 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lc8wg" event={"ID":"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c","Type":"ContainerStarted","Data":"142adb2914fb6f699dd4afa4facada1ac6c48357755fe975c8307b7239f0e634"} Oct 29 22:14:15.584003 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:15.583975 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="0406a95a779a6432af47d8ddd9a28a4a656f5d8c437e169524e8b5230df8760e" exitCode=0 Oct 29 22:14:15.584131 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:15.584081 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerDied","Data":"0406a95a779a6432af47d8ddd9a28a4a656f5d8c437e169524e8b5230df8760e"} Oct 29 22:14:15.617284 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:15.617233 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lc8wg" podStartSLOduration=3.080516108 podStartE2EDuration="11.617218711s" totalImagesPullingTime="8.536702603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:15.616903815 +0000 UTC m=+12.830254519" watchObservedRunningTime="2026-10-29 22:14:15.617218711 +0000 UTC m=+12.830569414" Oct 29 22:14:16.344011 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:16.343938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:16.344011 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:16.343964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:16.344256 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:16.344077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:16.344323 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:16.344252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:16.344382 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:16.344335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:16.344426 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:16.344411 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:16.587267 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:16.587233 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" event={"ID":"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0","Type":"ContainerStarted","Data":"ea4f859d44834702ca472a1e989de10347280c2c8842898d9e38b2f3c54947fa"} Oct 29 22:14:16.624411 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:16.624308 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" podStartSLOduration=2.567165278 podStartE2EDuration="13.624291287s" totalImagesPullingTime="11.057126009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:16.623944215 +0000 UTC m=+13.837294921" watchObservedRunningTime="2026-10-29 22:14:16.624291287 +0000 UTC m=+13.837641994" Oct 29 22:14:17.592555 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:17.592520 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="08ba8bef42f88f015121d06c5b25b0def6720e3781b961a1922e67b17a2fb723" exitCode=0 Oct 29 22:14:17.593123 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:17.592575 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerDied","Data":"08ba8bef42f88f015121d06c5b25b0def6720e3781b961a1922e67b17a2fb723"} Oct 29 22:14:18.344123 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:18.344091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:18.344310 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:18.344243 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:18.344629 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:18.344608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:18.344720 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:18.344701 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:18.344772 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:18.344748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:18.344829 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:18.344813 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:19.600071 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:19.599876 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="22fc1d9d0ae6fddd30897fbe9c9714b1d185cc2ba850743c4016b9decda30280" exitCode=0 Oct 29 22:14:19.600071 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:19.599938 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerDied","Data":"22fc1d9d0ae6fddd30897fbe9c9714b1d185cc2ba850743c4016b9decda30280"} Oct 29 22:14:20.041846 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.041818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:20.041958 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.041870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:20.041958 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.041954 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:20.042031 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.041957 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:20.042031 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.042000 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret podName:5f6c3496-906d-4d23-8e0c-a518b01f9ca3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.041987926 +0000 UTC m=+33.255338611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret") pod "global-pull-secret-syncer-px4td" (UID: "5f6c3496-906d-4d23-8e0c-a518b01f9ca3") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:20.042031 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.042014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.04200721 +0000 UTC m=+33.255357891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:20.142884 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.142859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:20.143021 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.143005 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:20.143075 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.143025 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:20.143075 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.143035 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xct6r for pod openshift-network-diagnostics/network-check-target-k2gxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:20.143189 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.143076 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r podName:93780727-5cdc-4eaa-91fc-92e7c3889eaa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.143064452 +0000 UTC m=+33.356415133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xct6r" (UniqueName: "kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r") pod "network-check-target-k2gxs" (UID: "93780727-5cdc-4eaa-91fc-92e7c3889eaa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:20.343635 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.343592 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:20.343635 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.343607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:20.343799 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.343698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:20.343799 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.343701 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:20.343799 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.343788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:20.343948 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:20.343885 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:20.603782 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.603758 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:20.604128 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.604106 2577 generic.go:356] "Generic (PLEG): container finished" podID="0762bfee-d0ac-4a88-bee5-212f3b230a2d" containerID="b65fca3491b518a929662565654683a53c108068414744c749ce694a9bcc69a6" exitCode=1 Oct 29 22:14:20.604208 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.604184 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"0e2305dc7ca9a349addbfb7dfd690d92ba232f44b55baabcf49cd33b6bf0c296"} Oct 29 22:14:20.604280 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.604266 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"b2bb614cf6f8d9a5330408db5b0a3cd28d4c2a47f44499b7ef4e83b442b7c7c3"} Oct 29 22:14:20.604312 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.604303 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerDied","Data":"b65fca3491b518a929662565654683a53c108068414744c749ce694a9bcc69a6"} Oct 29 22:14:20.604363 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.604338 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"9a0438de8338c367881613a3eb6fbe73ab8019f79b4fc4b8d8e1a1d630fe00d1"} Oct 29 22:14:20.607441 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.607415 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="69f7676af8ab1b0bace5871bca76c3bc3d6090dd16db75bd84bf18922b2d3681" exitCode=0 Oct 29 22:14:20.607531 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:20.607451 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerDied","Data":"69f7676af8ab1b0bace5871bca76c3bc3d6090dd16db75bd84bf18922b2d3681"} Oct 29 22:14:21.612578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:21.612402 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:21.613125 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:21.613099 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"23ce5437e0bfb00e4c8fd2d67449d5cdee91ae085579679bd96e1babea8c5593"} Oct 29 22:14:21.613198 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:21.613182 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"9457ec934c7fba09ef0442e3c4ef95f540e6269ba3bebf0b8d73308c5932d6aa"} Oct 29 22:14:21.616111 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:21.616091 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" event={"ID":"7c439b93-fb4b-441d-96d5-54852b26dcde","Type":"ContainerStarted","Data":"5ba9ca682535a3258d19d9c0c311a1cd2da06d18a57ea15f381464f7d44ff02c"} Oct 29 22:14:21.679890 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:21.679849 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t7vcg" podStartSLOduration=8.020862625 podStartE2EDuration="18.679839444s" totalImagesPullingTime="10.658976819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:21.679694871 +0000 UTC m=+18.893045574" watchObservedRunningTime="2026-10-29 22:14:21.679839444 +0000 UTC m=+18.893190143" Oct 29 22:14:22.094071 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:22.094040 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:22.094227 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:22.094171 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:22.094604 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:22.094587 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:22.343835 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:22.343807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:22.343964 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:22.343879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:22.343964 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:22.343957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:22.344061 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:22.343956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:22.344235 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:22.344082 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:22.344235 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:22.344186 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:23.623538 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:23.623506 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:23.624186 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:23.623864 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"990da4cf47a44c157edfb43303d3f4033958651a3292432afb965c916524c472"} Oct 29 22:14:24.344027 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:24.343999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:24.344223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:24.344081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:24.344223 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:24.344097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:24.344223 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:24.344208 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:24.344339 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:24.344278 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:24.344408 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:24.344377 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:25.630502 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:25.630476 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:25.631239 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:25.630788 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"0101903d4a9cf45a223be1abf9387f3cc6244916119ca36d72dc662578a58b83"} Oct 29 22:14:25.631239 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:25.631103 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:25.631239 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:25.631226 2577 scope.go:122] "RemoveContainer" containerID="b65fca3491b518a929662565654683a53c108068414744c749ce694a9bcc69a6" Oct 29 22:14:25.647525 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:25.647506 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:26.344477 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.344254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:26.344647 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.344254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:26.344647 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:26.344583 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:26.344647 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.344316 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:26.344758 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:26.344618 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:26.344800 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:26.344778 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:26.636435 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.636353 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:26.636830 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.636749 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" event={"ID":"0762bfee-d0ac-4a88-bee5-212f3b230a2d","Type":"ContainerStarted","Data":"638b934412df3bed626709e5d21e74ac2596ba0c7270c3d7ad1a5b6f713c7d30"} Oct 29 22:14:26.637079 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.637058 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:26.637241 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.637134 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:26.652018 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.651994 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:26.692544 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.692499 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" podStartSLOduration=7.134758047 podStartE2EDuration="22.692487675s" totalImagesPullingTime="15.557729628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:26.69169505 +0000 UTC m=+23.905045754" watchObservedRunningTime="2026-10-29 22:14:26.692487675 +0000 UTC m=+23.905838377" Oct 29 22:14:26.853036 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.853006 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-px4td"] Oct 29 22:14:26.853224 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.853109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:26.853295 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:26.853239 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:26.855981 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.855959 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8j2s2"] Oct 29 22:14:26.856098 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.856042 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:26.856162 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:26.856128 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:26.856469 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.856451 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k2gxs"] Oct 29 22:14:26.856536 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:26.856526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:26.856613 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:26.856598 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:28.343212 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:28.343185 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:28.343212 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:28.343211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:28.343728 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:28.343215 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:28.343728 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:28.343300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:28.343728 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:28.343352 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:28.343728 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:28.343411 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:30.343493 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:30.343453 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:30.343993 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:30.343469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:30.343993 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:30.343572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:30.343993 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:30.343565 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:30.343993 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:30.343643 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:30.343993 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:30.343726 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:32.343783 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:32.343747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:32.344212 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:32.343784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:32.344212 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:32.343864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8j2s2" podUID="d5414f15-670a-4ca7-a6c2-78a5426d717e" Oct 29 22:14:32.344212 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:32.343916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:32.344212 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:32.344001 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2gxs" podUID="93780727-5cdc-4eaa-91fc-92e7c3889eaa" Oct 29 22:14:32.344212 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:32.344074 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px4td" podUID="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" Oct 29 22:14:33.107273 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.107256 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeReady" Oct 29 22:14:33.107366 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.107343 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:14:33.147794 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.147725 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6vb55"] Oct 29 22:14:33.191085 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.191014 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vhdkj"] Oct 29 22:14:33.191860 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.191840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:33.202859 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.202840 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5p8jz"] Oct 29 22:14:33.203005 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.202977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:33.210799 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.208393 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.210799 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.208533 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.215394 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.215373 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pcdq8"] Oct 29 22:14:33.215515 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.215502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:33.233878 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.233859 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g"] Oct 29 22:14:33.234006 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.233984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.240437 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.240420 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b64c8c7d5-7zxkb"] Oct 29 22:14:33.240540 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.240497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.244899 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.244880 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5"] Oct 29 22:14:33.245030 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.245012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.246889 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.246872 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br"] Oct 29 22:14:33.247003 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.246966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.248994 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.248976 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 22:14:33.249329 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.249239 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-t6pf2"] Oct 29 22:14:33.249391 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.249366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.251982 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.251959 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4"] Oct 29 22:14:33.252096 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.252074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.256713 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.256604 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc"] Oct 29 22:14:33.256926 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.256908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.257022 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.257005 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 22:14:33.258759 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.258740 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 22:14:33.258898 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.258880 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 22:14:33.258967 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.258910 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 22:14:33.261804 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.261785 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk"] Oct 29 22:14:33.261919 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.261903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:33.265333 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.265311 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wsjxt"] Oct 29 22:14:33.265449 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.265411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:33.267276 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.267260 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6865b464d5-vpmdj"] Oct 29 22:14:33.267379 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.267358 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.270519 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.270498 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84d75cfc68-hctnk"] Oct 29 22:14:33.270643 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.270626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.274348 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.274332 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gxpdb"] Oct 29 22:14:33.274509 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.274408 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.278142 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.278124 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-97h69"] Oct 29 22:14:33.278252 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.278199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.282253 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282227 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6vb55"] Oct 29 22:14:33.282344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282259 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5p8jz"] Oct 29 22:14:33.282344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282280 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vhdkj"] Oct 29 22:14:33.282344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282300 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b64c8c7d5-7zxkb"] Oct 29 22:14:33.282344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282320 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4"] Oct 29 22:14:33.282344 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282339 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282344 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282359 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282380 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-t6pf2"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282399 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wsjxt"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282420 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282441 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pcdq8"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282460 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282481 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-97h69"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282500 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gxpdb"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282524 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282549 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6865b464d5-vpmdj"] Oct 29 22:14:33.282659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.282569 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84d75cfc68-hctnk"] Oct 29 22:14:33.311088 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311070 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.311354 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311337 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 22:14:33.311595 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311400 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.311595 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311405 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.311595 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311409 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.311595 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311539 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.311844 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311618 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.311844 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311699 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.311844 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311721 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311898 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311901 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311930 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311953 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311987 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311993 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.311998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311958 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.311993 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312031 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312074 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312087 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312135 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312169 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312177 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312183 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 22:14:33.312307 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312220 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.312695 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312311 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 22:14:33.312695 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312388 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.312695 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.312435 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:33.325412 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.325391 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 22:14:33.336087 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.336067 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 22:14:33.336578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.336561 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 22:14:33.336645 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.336582 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 22:14:33.339468 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhf5\" (UniqueName: \"kubernetes.io/projected/f52c0b21-62cc-4ae5-b309-a79f19fa8974-kube-api-access-wzhf5\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.339522 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be7200fc-4e5d-416b-8898-2a8bac5648aa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.339522 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.339522 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndm85\" (UniqueName: \"kubernetes.io/projected/ce594888-8263-4ccd-b24d-66900da5d257-kube-api-access-ndm85\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.339656 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbbl\" (UniqueName: \"kubernetes.io/projected/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c-kube-api-access-5zbbl\") pod \"kas-connection-checker-685895c7c9-6vb55\" (UID: \"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c\") " pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:33.339656 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.339717 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-stats-auth\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.339717 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.339717 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f52c0b21-62cc-4ae5-b309-a79f19fa8974-config\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.339847 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmljm\" (UniqueName: \"kubernetes.io/projected/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-kube-api-access-vmljm\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.339878 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27kwm\" (UniqueName: \"kubernetes.io/projected/c2f1152f-ec5b-4fb7-8b17-04396337eb07-kube-api-access-27kwm\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.339878 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-service-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.339958 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce594888-8263-4ccd-b24d-66900da5d257-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.339958 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f52c0b21-62cc-4ae5-b309-a79f19fa8974-serving-cert\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.339958 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.339958 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.340094 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.339982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.340094 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfmp\" (UniqueName: \"kubernetes.io/projected/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-kube-api-access-vpfmp\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.340158 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-tmp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.340158 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdjr\" (UniqueName: \"kubernetes.io/projected/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3-kube-api-access-2wdjr\") pod \"network-check-source-6f78d59f99-9xtxc\" (UID: \"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:33.340227 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-default-certificate\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.340227 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68d5z\" (UniqueName: \"kubernetes.io/projected/60db2db0-73c9-4cb7-aa1b-9c670677bc6e-kube-api-access-68d5z\") pod \"kas-connection-checker-685895c7c9-vhdkj\" (UID: \"60db2db0-73c9-4cb7-aa1b-9c670677bc6e\") " pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:33.340313 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9vj\" (UniqueName: \"kubernetes.io/projected/4d78fd0a-b2a6-4688-9435-1e5c972575f8-kube-api-access-vk9vj\") pod \"kas-connection-checker-685895c7c9-5p8jz\" (UID: \"4d78fd0a-b2a6-4688-9435-1e5c972575f8\") " pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:33.340313 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.340395 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e66c05-354b-4882-97a5-a3c247a548e8-serving-cert\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.340395 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce594888-8263-4ccd-b24d-66900da5d257-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.340395 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.340523 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pklpp\" (UniqueName: \"kubernetes.io/projected/f8e66c05-354b-4882-97a5-a3c247a548e8-kube-api-access-pklpp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.340523 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.340452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-snapshots\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.349681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349661 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349670 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349714 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349760 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349764 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349778 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349932 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 22:14:33.349990 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.349984 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 22:14:33.350265 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.350024 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 22:14:33.350265 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.350040 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qnnrl\"" Oct 29 22:14:33.350265 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.350031 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-558gk\"" Oct 29 22:14:33.350265 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.350028 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 22:14:33.350265 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.350101 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h6g72\"" Oct 29 22:14:33.441472 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-tmp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.441472 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.441472 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdjr\" (UniqueName: \"kubernetes.io/projected/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3-kube-api-access-2wdjr\") pod \"network-check-source-6f78d59f99-9xtxc\" (UID: \"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-default-certificate\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68d5z\" (UniqueName: \"kubernetes.io/projected/60db2db0-73c9-4cb7-aa1b-9c670677bc6e-kube-api-access-68d5z\") pod \"kas-connection-checker-685895c7c9-vhdkj\" (UID: \"60db2db0-73c9-4cb7-aa1b-9c670677bc6e\") " pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9vj\" (UniqueName: \"kubernetes.io/projected/4d78fd0a-b2a6-4688-9435-1e5c972575f8-kube-api-access-vk9vj\") pod \"kas-connection-checker-685895c7c9-5p8jz\" (UID: \"4d78fd0a-b2a6-4688-9435-1e5c972575f8\") " pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e66c05-354b-4882-97a5-a3c247a548e8-serving-cert\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce594888-8263-4ccd-b24d-66900da5d257-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.441697 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklpp\" (UniqueName: \"kubernetes.io/projected/f8e66c05-354b-4882-97a5-a3c247a548e8-kube-api-access-pklpp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-config-volume\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441847 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-tmp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01ed9874-d3cc-4b95-a198-348f68659495-serving-cert\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.441967 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:33.442001 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.441990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.442386 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.442386 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-snapshots\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.442386 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.442246 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:33.942226516 +0000 UTC m=+31.155577211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442529 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhf5\" (UniqueName: \"kubernetes.io/projected/f52c0b21-62cc-4ae5-b309-a79f19fa8974-kube-api-access-wzhf5\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-tmp-dir\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be7200fc-4e5d-416b-8898-2a8bac5648aa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442810 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-snapshots\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndm85\" (UniqueName: \"kubernetes.io/projected/ce594888-8263-4ccd-b24d-66900da5d257-kube-api-access-ndm85\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbbl\" (UniqueName: \"kubernetes.io/projected/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c-kube-api-access-5zbbl\") pod \"kas-connection-checker-685895c7c9-6vb55\" (UID: \"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c\") " pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e74b42e-7da7-4a53-817b-5a8873b3377d-cert\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.442986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-stats-auth\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.443474 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rfk\" (UniqueName: \"kubernetes.io/projected/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-kube-api-access-t7rfk\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f52c0b21-62cc-4ae5-b309-a79f19fa8974-config\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.443239 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:33.943220105 +0000 UTC m=+31.156570805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmljm\" (UniqueName: \"kubernetes.io/projected/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-kube-api-access-vmljm\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-config\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kwm\" (UniqueName: \"kubernetes.io/projected/c2f1152f-ec5b-4fb7-8b17-04396337eb07-kube-api-access-27kwm\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-service-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443501 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be7200fc-4e5d-416b-8898-2a8bac5648aa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce594888-8263-4ccd-b24d-66900da5d257-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-trusted-ca\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.444288 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.443991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-service-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.444921 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.444397 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce594888-8263-4ccd-b24d-66900da5d257-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.444921 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.444624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f52c0b21-62cc-4ae5-b309-a79f19fa8974-serving-cert\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.444921 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.444725 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.444921 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.444724 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f52c0b21-62cc-4ae5-b309-a79f19fa8974-config\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.445119 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.444921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbqh\" (UniqueName: \"kubernetes.io/projected/2e74b42e-7da7-4a53-817b-5a8873b3377d-kube-api-access-8pbqh\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.445119 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.444992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.445119 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.445113 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.445170 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.445260 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:33.945241909 +0000 UTC m=+31.158592602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.445286 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:33.945269432 +0000 UTC m=+31.158620126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : secret "router-metrics-certs-default" not found Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.445433 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.445423 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.445469 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:33.945456692 +0000 UTC m=+31.158807372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfmp\" (UniqueName: \"kubernetes.io/projected/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-kube-api-access-vpfmp\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mwx\" (UniqueName: \"kubernetes.io/projected/01ed9874-d3cc-4b95-a198-348f68659495-kube-api-access-m8mwx\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4278f\" (UniqueName: \"kubernetes.io/projected/a1cc6e96-33dc-4727-b69a-c3050aee425c-kube-api-access-4278f\") pod \"volume-data-source-validator-58c9f5f46d-s4rsk\" (UID: \"a1cc6e96-33dc-4727-b69a-c3050aee425c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.445837 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.445673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.446486 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.446210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce594888-8263-4ccd-b24d-66900da5d257-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.446548 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.446512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e66c05-354b-4882-97a5-a3c247a548e8-serving-cert\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.447919 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.447883 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-stats-auth\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.448006 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.447956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-default-certificate\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.448457 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.448435 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f52c0b21-62cc-4ae5-b309-a79f19fa8974-serving-cert\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.489620 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.489597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdjr\" (UniqueName: \"kubernetes.io/projected/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3-kube-api-access-2wdjr\") pod \"network-check-source-6f78d59f99-9xtxc\" (UID: \"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:33.489853 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.489830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklpp\" (UniqueName: \"kubernetes.io/projected/f8e66c05-354b-4882-97a5-a3c247a548e8-kube-api-access-pklpp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.491951 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.491728 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfmp\" (UniqueName: \"kubernetes.io/projected/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-kube-api-access-vpfmp\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.491951 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.491894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9vj\" (UniqueName: \"kubernetes.io/projected/4d78fd0a-b2a6-4688-9435-1e5c972575f8-kube-api-access-vk9vj\") pod \"kas-connection-checker-685895c7c9-5p8jz\" (UID: \"4d78fd0a-b2a6-4688-9435-1e5c972575f8\") " pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:33.491951 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.491894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d5z\" (UniqueName: \"kubernetes.io/projected/60db2db0-73c9-4cb7-aa1b-9c670677bc6e-kube-api-access-68d5z\") pod \"kas-connection-checker-685895c7c9-vhdkj\" (UID: \"60db2db0-73c9-4cb7-aa1b-9c670677bc6e\") " pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:33.492103 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.492043 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndm85\" (UniqueName: \"kubernetes.io/projected/ce594888-8263-4ccd-b24d-66900da5d257-kube-api-access-ndm85\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.492331 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.492307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kwm\" (UniqueName: \"kubernetes.io/projected/c2f1152f-ec5b-4fb7-8b17-04396337eb07-kube-api-access-27kwm\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.492372 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.492329 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhf5\" (UniqueName: \"kubernetes.io/projected/f52c0b21-62cc-4ae5-b309-a79f19fa8974-kube-api-access-wzhf5\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.492372 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.492363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmljm\" (UniqueName: \"kubernetes.io/projected/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-kube-api-access-vmljm\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.493360 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.493345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbbl\" (UniqueName: \"kubernetes.io/projected/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c-kube-api-access-5zbbl\") pod \"kas-connection-checker-685895c7c9-6vb55\" (UID: \"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c\") " pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:33.515309 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.515287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:33.532376 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.532359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:33.536913 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.536899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:33.546455 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.546526 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-config-volume\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.546526 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.546526 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01ed9874-d3cc-4b95-a198-348f68659495-serving-cert\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.546681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.546681 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.546780 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.546780 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-tmp-dir\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.546780 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.546923 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e74b42e-7da7-4a53-817b-5a8873b3377d-cert\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.546923 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.546923 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.546923 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.546923 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rfk\" (UniqueName: \"kubernetes.io/projected/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-kube-api-access-t7rfk\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.546974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-config\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-trusted-ca\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbqh\" (UniqueName: \"kubernetes.io/projected/2e74b42e-7da7-4a53-817b-5a8873b3377d-kube-api-access-8pbqh\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547084 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-config-volume\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.547204 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mwx\" (UniqueName: \"kubernetes.io/projected/01ed9874-d3cc-4b95-a198-348f68659495-kube-api-access-m8mwx\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4278f\" (UniqueName: \"kubernetes.io/projected/a1cc6e96-33dc-4727-b69a-c3050aee425c-kube-api-access-4278f\") pod \"volume-data-source-validator-58c9f5f46d-s4rsk\" (UID: \"a1cc6e96-33dc-4727-b69a-c3050aee425c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547473 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547485 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6865b464d5-vpmdj: secret "image-registry-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.547521 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-tmp-dir\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547529 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls podName:a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.047514228 +0000 UTC m=+31.260864910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls") pod "image-registry-6865b464d5-vpmdj" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00") : secret "image-registry-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547543 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547605 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls podName:edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.047588783 +0000 UTC m=+31.260939471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls") pod "dns-default-gxpdb" (UID: "edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3") : secret "dns-default-metrics-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547614 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547629 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d75cfc68-hctnk: secret "image-registry-tls" not found Oct 29 22:14:33.547888 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.547687 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls podName:78c2f084-7af8-4fe9-824a-2e50a58eae25 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.047655645 +0000 UTC m=+31.261006358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls") pod "image-registry-84d75cfc68-hctnk" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25") : secret "image-registry-tls" not found Oct 29 22:14:33.548621 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.548280 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.548621 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.548280 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.548621 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.548329 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-config\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.548793 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.548771 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.548854 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.548820 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.549811 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.548873 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-trusted-ca\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.549811 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.549661 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.550589 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.550566 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.550944 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.550918 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01ed9874-d3cc-4b95-a198-348f68659495-serving-cert\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.551033 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.550986 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.551249 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.551226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.551690 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.551670 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e74b42e-7da7-4a53-817b-5a8873b3377d-cert\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.554414 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.554396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:33.568186 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.568129 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.569217 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.569194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbqh\" (UniqueName: \"kubernetes.io/projected/2e74b42e-7da7-4a53-817b-5a8873b3377d-kube-api-access-8pbqh\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.569313 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.569247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rfk\" (UniqueName: \"kubernetes.io/projected/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-kube-api-access-t7rfk\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:33.569574 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.569554 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4278f\" (UniqueName: \"kubernetes.io/projected/a1cc6e96-33dc-4727-b69a-c3050aee425c-kube-api-access-4278f\") pod \"volume-data-source-validator-58c9f5f46d-s4rsk\" (UID: \"a1cc6e96-33dc-4727-b69a-c3050aee425c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:33.569897 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.569872 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:33.569974 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.569900 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.570208 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.570193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:33.571225 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.571209 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mwx\" (UniqueName: \"kubernetes.io/projected/01ed9874-d3cc-4b95-a198-348f68659495-kube-api-access-m8mwx\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.584578 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.584549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:33.595011 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.594985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:33.603521 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.603497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:33.607668 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.607645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:33.609787 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.609768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:33.672367 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.672341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:33.883324 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.883301 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vhdkj"] Oct 29 22:14:33.885777 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.885751 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6vb55"] Oct 29 22:14:33.888132 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:33.888110 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60db2db0_73c9_4cb7_aa1b_9c670677bc6e.slice/crio-a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd WatchSource:0}: Error finding container a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd: Status 404 returned error can't find the container with id a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd Oct 29 22:14:33.890130 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:33.890113 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf5abc1_44fb_4ec3_b7f7_f9e4b866a09c.slice/crio-d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c WatchSource:0}: Error finding container d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c: Status 404 returned error can't find the container with id d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c Oct 29 22:14:33.933313 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.933240 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g"] Oct 29 22:14:33.941263 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.941129 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5p8jz"] Oct 29 22:14:33.942203 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:33.942168 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52c0b21_62cc_4ae5_b309_a79f19fa8974.slice/crio-04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d WatchSource:0}: Error finding container 04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d: Status 404 returned error can't find the container with id 04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d Oct 29 22:14:33.947576 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:33.947551 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d78fd0a_b2a6_4688_9435_1e5c972575f8.slice/crio-b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc WatchSource:0}: Error finding container b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc: Status 404 returned error can't find the container with id b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.951442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.951528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.951580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.951606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.951646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.951790 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.951840 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.951822593 +0000 UTC m=+32.165173275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.951901 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.951935 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.951924378 +0000 UTC m=+32.165275072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.952013 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.952004098 +0000 UTC m=+32.165354785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.952067 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.952100 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.952088953 +0000 UTC m=+32.165439636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : secret "router-metrics-certs-default" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.952173 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:33.952511 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:33.952206 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.952196247 +0000 UTC m=+32.165547137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:33.977619 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:33.977598 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-t6pf2"] Oct 29 22:14:33.986656 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:33.986624 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e66c05_354b_4882_97a5_a3c247a548e8.slice/crio-c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53 WatchSource:0}: Error finding container c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53: Status 404 returned error can't find the container with id c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53 Oct 29 22:14:34.020410 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.020383 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4"] Oct 29 22:14:34.023334 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.023313 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc"] Oct 29 22:14:34.024447 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:34.024424 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce594888_8263_4ccd_b24d_66900da5d257.slice/crio-637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41 WatchSource:0}: Error finding container 637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41: Status 404 returned error can't find the container with id 637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41 Oct 29 22:14:34.026103 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.026078 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wsjxt"] Oct 29 22:14:34.035030 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:34.035004 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547cbb0b_f75f_4a5d_80ee_4d7073cc76f3.slice/crio-181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8 WatchSource:0}: Error finding container 181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8: Status 404 returned error can't find the container with id 181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8 Oct 29 22:14:34.035273 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:34.035255 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ed9874_d3cc_4b95_a198_348f68659495.slice/crio-40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05 WatchSource:0}: Error finding container 40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05: Status 404 returned error can't find the container with id 40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05 Oct 29 22:14:34.049298 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.049275 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk"] Oct 29 22:14:34.052170 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.052126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:34.052390 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052367 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:34.052390 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052384 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6865b464d5-vpmdj: secret "image-registry-tls" not found Oct 29 22:14:34.052505 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.052392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:34.052505 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052428 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls podName:a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:35.052412 +0000 UTC m=+32.265762682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls") pod "image-registry-6865b464d5-vpmdj" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00") : secret "image-registry-tls" not found Oct 29 22:14:34.052505 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.052473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:34.052505 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052495 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:34.052695 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052510 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d75cfc68-hctnk: secret "image-registry-tls" not found Oct 29 22:14:34.052695 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052558 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls podName:78c2f084-7af8-4fe9-824a-2e50a58eae25 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:35.052542897 +0000 UTC m=+32.265893583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls") pod "image-registry-84d75cfc68-hctnk" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25") : secret "image-registry-tls" not found Oct 29 22:14:34.052695 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052594 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:14:34.052695 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.052644 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls podName:edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:35.052632504 +0000 UTC m=+32.265983185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls") pod "dns-default-gxpdb" (UID: "edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3") : secret "dns-default-metrics-tls" not found Oct 29 22:14:34.053899 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:34.053872 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cc6e96_33dc_4727_b69a_c3050aee425c.slice/crio-2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2 WatchSource:0}: Error finding container 2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2: Status 404 returned error can't find the container with id 2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2 Oct 29 22:14:34.080007 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.079983 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-97h69"] Oct 29 22:14:34.083878 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:34.083858 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e74b42e_7da7_4a53_817b_5a8873b3377d.slice/crio-6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b WatchSource:0}: Error finding container 6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b: Status 404 returned error can't find the container with id 6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b Oct 29 22:14:34.343473 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.343449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:34.343571 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.343476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:34.343632 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.343596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:34.361849 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.361832 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:14:34.362184 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.361975 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:14:34.362184 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.361999 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:14:34.362184 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.362032 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:14:34.362184 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.362061 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:14:34.659733 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.659609 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" event={"ID":"01ed9874-d3cc-4b95-a198-348f68659495","Type":"ContainerStarted","Data":"40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05"} Oct 29 22:14:34.662333 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.662297 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" event={"ID":"f8e66c05-354b-4882-97a5-a3c247a548e8","Type":"ContainerStarted","Data":"c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53"} Oct 29 22:14:34.664055 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.664030 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" event={"ID":"ce594888-8263-4ccd-b24d-66900da5d257","Type":"ContainerStarted","Data":"637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41"} Oct 29 22:14:34.666279 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.666253 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" event={"ID":"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3","Type":"ContainerStarted","Data":"181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8"} Oct 29 22:14:34.667944 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.667919 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" event={"ID":"f52c0b21-62cc-4ae5-b309-a79f19fa8974","Type":"ContainerStarted","Data":"04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d"} Oct 29 22:14:34.671100 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.671075 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6vb55" event={"ID":"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c","Type":"ContainerStarted","Data":"7fc951094e7e50391ef0dd6080db213188dc1efb3ebc99f6236373e686fbde2c"} Oct 29 22:14:34.671233 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.671123 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6vb55" event={"ID":"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c","Type":"ContainerStarted","Data":"d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c"} Oct 29 22:14:34.673838 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.673461 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" event={"ID":"4d78fd0a-b2a6-4688-9435-1e5c972575f8","Type":"ContainerStarted","Data":"46d6659bc3773c7742289dbd741adb1cf925982b1bf8b62a02697c94a6e290b7"} Oct 29 22:14:34.673838 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.673501 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" event={"ID":"4d78fd0a-b2a6-4688-9435-1e5c972575f8","Type":"ContainerStarted","Data":"b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc"} Oct 29 22:14:34.676320 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.675844 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" event={"ID":"60db2db0-73c9-4cb7-aa1b-9c670677bc6e","Type":"ContainerStarted","Data":"88abdd5a20d2ddb76b66bc7f79c2e12fec686ae345ed4567db41dc49f280bc08"} Oct 29 22:14:34.676320 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.675884 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" event={"ID":"60db2db0-73c9-4cb7-aa1b-9c670677bc6e","Type":"ContainerStarted","Data":"a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd"} Oct 29 22:14:34.677880 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.677849 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-97h69" event={"ID":"2e74b42e-7da7-4a53-817b-5a8873b3377d","Type":"ContainerStarted","Data":"6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b"} Oct 29 22:14:34.679617 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.679563 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" event={"ID":"a1cc6e96-33dc-4727-b69a-c3050aee425c","Type":"ContainerStarted","Data":"2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2"} Oct 29 22:14:34.709620 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.709560 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6vb55" podStartSLOduration=371.709543943 podStartE2EDuration="6m11.709543943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.707587862 +0000 UTC m=+31.920938568" watchObservedRunningTime="2026-10-29 22:14:34.709543943 +0000 UTC m=+31.922894647" Oct 29 22:14:34.744021 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.743965 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" podStartSLOduration=371.743948346 podStartE2EDuration="6m11.743948346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.742609897 +0000 UTC m=+31.955960601" watchObservedRunningTime="2026-10-29 22:14:34.743948346 +0000 UTC m=+31.957299054" Oct 29 22:14:34.961043 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.961007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:34.961234 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.961059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:34.961234 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.961103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:34.961234 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.961208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:34.961394 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:34.961295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961451 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.961433581 +0000 UTC m=+34.174784268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961548 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961603 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.961585121 +0000 UTC m=+34.174935819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961651 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961675 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.961667284 +0000 UTC m=+34.175018002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : secret "router-metrics-certs-default" not found Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961719 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961779 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.961740569 +0000 UTC m=+34.175091276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:34.961842 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961815 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:34.962231 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:34.961865 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.961850257 +0000 UTC m=+34.175200946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:35.062241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:35.062281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:35.062328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062452 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062466 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6865b464d5-vpmdj: secret "image-registry-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062514 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls podName:a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:37.062498604 +0000 UTC m=+34.275849298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls") pod "image-registry-6865b464d5-vpmdj" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00") : secret "image-registry-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062835 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062849 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d75cfc68-hctnk: secret "image-registry-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062891 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls podName:78c2f084-7af8-4fe9-824a-2e50a58eae25 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:37.062876178 +0000 UTC m=+34.276226864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls") pod "image-registry-84d75cfc68-hctnk" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25") : secret "image-registry-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062954 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:14:35.063052 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:35.062986 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls podName:edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:37.062974995 +0000 UTC m=+34.276325679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls") pod "dns-default-gxpdb" (UID: "edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3") : secret "dns-default-metrics-tls" not found Oct 29 22:14:36.071562 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.070711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:36.071562 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.070769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:36.071562 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.070916 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 22:14:36.071562 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.070991 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs podName:d5414f15-670a-4ca7-a6c2-78a5426d717e nodeName:}" failed. No retries permitted until 2026-10-29 22:15:08.0709723 +0000 UTC m=+65.284322995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs") pod "network-metrics-daemon-8j2s2" (UID: "d5414f15-670a-4ca7-a6c2-78a5426d717e") : secret "metrics-daemon-secret" not found Oct 29 22:14:36.086217 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.077318 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:36.168853 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.168636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:36.172127 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.171634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:36.176708 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.176683 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:36.223591 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.223411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:36.490936 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.489522 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" podStartSLOduration=373.489502167 podStartE2EDuration="6m13.489502167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.778688623 +0000 UTC m=+31.992039327" watchObservedRunningTime="2026-10-29 22:14:36.489502167 +0000 UTC m=+33.702852871" Oct 29 22:14:36.490936 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.490742 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-px4td"] Oct 29 22:14:36.497081 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:36.497043 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6c3496_906d_4d23_8e0c_a518b01f9ca3.slice/crio-f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3 WatchSource:0}: Error finding container f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3: Status 404 returned error can't find the container with id f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3 Oct 29 22:14:36.586207 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.586173 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k2gxs"] Oct 29 22:14:36.691623 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.691587 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k2gxs" event={"ID":"93780727-5cdc-4eaa-91fc-92e7c3889eaa","Type":"ContainerStarted","Data":"879cec33bead00a540eaa87c45440fe3d155f754a197e880de4a32b7719dccb6"} Oct 29 22:14:36.694085 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.694027 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-px4td" event={"ID":"5f6c3496-906d-4d23-8e0c-a518b01f9ca3","Type":"ContainerStarted","Data":"f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3"} Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.980909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.980955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.981000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.981087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:36.981169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981334 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.981301361 +0000 UTC m=+38.194652057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981731 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981783 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.981767172 +0000 UTC m=+38.195117867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : secret "router-metrics-certs-default" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981845 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981880 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.981869037 +0000 UTC m=+38.195219721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981941 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.981974 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.981962833 +0000 UTC m=+38.195313528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.982029 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:36.982181 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:36.982057 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.982047895 +0000 UTC m=+38.195398582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:37.082347 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:37.081624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:37.082347 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:37.081680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:37.082347 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:37.081724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:37.082347 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.081917 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:37.082347 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.081932 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6865b464d5-vpmdj: secret "image-registry-tls" not found Oct 29 22:14:37.082347 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.081989 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls podName:a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:41.081970679 +0000 UTC m=+38.295321362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls") pod "image-registry-6865b464d5-vpmdj" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00") : secret "image-registry-tls" not found Oct 29 22:14:37.083703 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.082642 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:37.083703 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.082764 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d75cfc68-hctnk: secret "image-registry-tls" not found Oct 29 22:14:37.083703 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.082825 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls podName:78c2f084-7af8-4fe9-824a-2e50a58eae25 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:41.082799961 +0000 UTC m=+38.296150649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls") pod "image-registry-84d75cfc68-hctnk" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25") : secret "image-registry-tls" not found Oct 29 22:14:37.083703 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.082973 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:14:37.083703 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:37.083024 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls podName:edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:41.083010051 +0000 UTC m=+38.296360738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls") pod "dns-default-gxpdb" (UID: "edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3") : secret "dns-default-metrics-tls" not found Oct 29 22:14:37.703224 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:37.702442 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" event={"ID":"a1cc6e96-33dc-4727-b69a-c3050aee425c","Type":"ContainerStarted","Data":"6efa19f7be4977c19a83e061760225ced066e496b7b509bc47d037108fbdb0bd"} Oct 29 22:14:37.739764 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:37.739705 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" podStartSLOduration=343.94868704 podStartE2EDuration="5m46.739686822s" totalImagesPullingTime="2.790999782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:37.738982123 +0000 UTC m=+34.952332826" watchObservedRunningTime="2026-10-29 22:14:37.739686822 +0000 UTC m=+34.953037525" Oct 29 22:14:38.709575 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:38.708334 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/0.log" Oct 29 22:14:38.709575 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:38.708377 2577 generic.go:356] "Generic (PLEG): container finished" podID="01ed9874-d3cc-4b95-a198-348f68659495" containerID="acc97c2e1eb207d726a256be07782cc7287f852b7455fe06fd321b42a20a42c2" exitCode=255 Oct 29 22:14:38.709575 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:38.709055 2577 scope.go:122] "RemoveContainer" containerID="acc97c2e1eb207d726a256be07782cc7287f852b7455fe06fd321b42a20a42c2" Oct 29 22:14:38.709575 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:38.709311 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" event={"ID":"01ed9874-d3cc-4b95-a198-348f68659495","Type":"ContainerDied","Data":"acc97c2e1eb207d726a256be07782cc7287f852b7455fe06fd321b42a20a42c2"} Oct 29 22:14:39.575703 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.575665 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p6ccn"] Oct 29 22:14:39.601360 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.601333 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p6ccn"] Oct 29 22:14:39.601517 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.601464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.621464 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.621254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69wgp\" (UniqueName: \"kubernetes.io/projected/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-api-access-69wgp\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.621572 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.621552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd4aa13b-38c8-4806-978a-0ce17607e28b-crio-socket\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.621645 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.621611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.621693 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.621668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd4aa13b-38c8-4806-978a-0ce17607e28b-data-volume\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.621739 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.621700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.625773 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.625674 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:14:39.625773 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.625730 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:14:39.626205 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.626184 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9x6m\"" Oct 29 22:14:39.656946 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.656921 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx"] Oct 29 22:14:39.674434 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.674413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.679984 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.679961 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx"] Oct 29 22:14:39.719075 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.719033 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" event={"ID":"f52c0b21-62cc-4ae5-b309-a79f19fa8974","Type":"ContainerStarted","Data":"978faa7b7caea8b2056adb61add983d0dd22fa31613814652828b17cae2fbe15"} Oct 29 22:14:39.721047 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.721029 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pdxn8\"" Oct 29 22:14:39.722105 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.722222 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wgp\" (UniqueName: \"kubernetes.io/projected/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-api-access-69wgp\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.722222 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.722222 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.722325 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd4aa13b-38c8-4806-978a-0ce17607e28b-crio-socket\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.722325 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.722325 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.722417 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd4aa13b-38c8-4806-978a-0ce17607e28b-data-volume\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.722417 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.722478 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:39.722433 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:14:39.722478 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:39.722473 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.222460612 +0000 UTC m=+37.435811293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : configmap "kube-rbac-proxy" not found Oct 29 22:14:39.722945 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.722912 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd4aa13b-38c8-4806-978a-0ce17607e28b-crio-socket\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.723005 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:39.722987 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:39.723046 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:39.723020 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:40.223010145 +0000 UTC m=+37.436360826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:39.723313 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.723256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd4aa13b-38c8-4806-978a-0ce17607e28b-data-volume\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.726452 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.726436 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/1.log" Oct 29 22:14:39.727057 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.727041 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/0.log" Oct 29 22:14:39.727137 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.727069 2577 generic.go:356] "Generic (PLEG): container finished" podID="01ed9874-d3cc-4b95-a198-348f68659495" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" exitCode=255 Oct 29 22:14:39.727137 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.727113 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" event={"ID":"01ed9874-d3cc-4b95-a198-348f68659495","Type":"ContainerDied","Data":"aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1"} Oct 29 22:14:39.727316 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.727144 2577 scope.go:122] "RemoveContainer" containerID="acc97c2e1eb207d726a256be07782cc7287f852b7455fe06fd321b42a20a42c2" Oct 29 22:14:39.727440 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.727420 2577 scope.go:122] "RemoveContainer" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" Oct 29 22:14:39.727590 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:39.727569 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wsjxt_openshift-console-operator(01ed9874-d3cc-4b95-a198-348f68659495)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" Oct 29 22:14:39.730048 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.730007 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" event={"ID":"f8e66c05-354b-4882-97a5-a3c247a548e8","Type":"ContainerStarted","Data":"e7e1a78302b29250ddf2957583e5551ed96d33184a7db9c3a0327666f417f669"} Oct 29 22:14:39.748296 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.748246 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69wgp\" (UniqueName: \"kubernetes.io/projected/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-api-access-69wgp\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:39.755919 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.755878 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" podStartSLOduration=366.637336688 podStartE2EDuration="6m11.755867701s" totalImagesPullingTime="5.118531013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:39.754890061 +0000 UTC m=+36.968240765" watchObservedRunningTime="2026-10-29 22:14:39.755867701 +0000 UTC m=+36.969218403" Oct 29 22:14:39.824007 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.823198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.824007 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.823271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.824007 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.823386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.824007 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.823565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.825749 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.825047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.825749 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.825326 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.825825 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.825749 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:39.855869 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:39.855558 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:40.019906 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.019864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:40.229684 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.229061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:40.229684 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.229125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:40.229684 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:40.229251 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:14:40.229684 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:40.229300 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:41.229286724 +0000 UTC m=+38.442637409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : configmap "kube-rbac-proxy" not found Oct 29 22:14:40.229684 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:40.229617 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:40.229684 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:40.229653 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:41.229643208 +0000 UTC m=+38.442993888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:40.369027 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.367482 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" podStartSLOduration=344.723189921 podStartE2EDuration="5m49.367463519s" totalImagesPullingTime="4.644273598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:39.830775766 +0000 UTC m=+37.044126466" watchObservedRunningTime="2026-10-29 22:14:40.367463519 +0000 UTC m=+37.580814227" Oct 29 22:14:40.369027 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.368886 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx"] Oct 29 22:14:40.736418 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.736338 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" event={"ID":"ce594888-8263-4ccd-b24d-66900da5d257","Type":"ContainerStarted","Data":"5088e0bf1667b133600a8adb1d80d2501294863e837430277c5c242d7f73668d"} Oct 29 22:14:40.739106 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.739069 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-97h69" event={"ID":"2e74b42e-7da7-4a53-817b-5a8873b3377d","Type":"ContainerStarted","Data":"3d7b2b322665029b276c77a2558084e9a333a1a87fef26395e17eb8817a9760c"} Oct 29 22:14:40.741937 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.741595 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/1.log" Oct 29 22:14:40.742143 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.742121 2577 scope.go:122] "RemoveContainer" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" Oct 29 22:14:40.742653 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:40.742338 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wsjxt_openshift-console-operator(01ed9874-d3cc-4b95-a198-348f68659495)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" Oct 29 22:14:40.745784 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.745679 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" event={"ID":"f7f1df3e-2020-4541-8b82-363ccdad9a05","Type":"ContainerStarted","Data":"1ce7b589dc1e7a1f1248ee320607fea48cf57512ef9b2493d7e9d1d9dae24374"} Oct 29 22:14:40.745784 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.745719 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" event={"ID":"f7f1df3e-2020-4541-8b82-363ccdad9a05","Type":"ContainerStarted","Data":"4cda4a472c66dfdd7fb1a0fe509e5fd44f020e62cd8fe1e90a3cc9d621be4663"} Oct 29 22:14:40.788556 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.788507 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" podStartSLOduration=344.170789138 podStartE2EDuration="5m49.78849448s" totalImagesPullingTime="5.617705342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:40.775009442 +0000 UTC m=+37.988360149" watchObservedRunningTime="2026-10-29 22:14:40.78849448 +0000 UTC m=+38.001845183" Oct 29 22:14:40.789247 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.789224 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-25css"] Oct 29 22:14:40.806239 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.805864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.812690 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.812664 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-25css"] Oct 29 22:14:40.820309 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.820285 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 22:14:40.820445 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.820337 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 22:14:40.820445 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.820369 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 22:14:40.820445 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.820413 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 22:14:40.820695 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.820450 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fw4tl\"" Oct 29 22:14:40.837185 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.836360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-key\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.837185 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.836420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-cabundle\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.837185 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.836526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bww45\" (UniqueName: \"kubernetes.io/projected/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-kube-api-access-bww45\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.853369 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.852887 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" podStartSLOduration=1.852871081 podStartE2EDuration="1.852871081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:40.851724254 +0000 UTC m=+38.065074959" watchObservedRunningTime="2026-10-29 22:14:40.852871081 +0000 UTC m=+38.066221785" Oct 29 22:14:40.890349 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.890283 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-97h69" podStartSLOduration=1.9522596399999999 podStartE2EDuration="7.89026776s" totalImagesPullingTime="5.93800812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:40.888395384 +0000 UTC m=+38.101746088" watchObservedRunningTime="2026-10-29 22:14:40.89026776 +0000 UTC m=+38.103618464" Oct 29 22:14:40.937749 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.937421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-key\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.937749 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.937466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-cabundle\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.937749 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.937535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww45\" (UniqueName: \"kubernetes.io/projected/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-kube-api-access-bww45\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.938819 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.938785 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-cabundle\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.947994 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.947968 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-key\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:40.965565 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:40.965535 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww45\" (UniqueName: \"kubernetes.io/projected/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-kube-api-access-bww45\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:41.038848 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.038781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:41.038980 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.038864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:41.038980 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.038921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:41.038980 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.038933 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:41.038980 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.038946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.038992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.038988719 +0000 UTC m=+46.252339410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039043 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.039024248 +0000 UTC m=+46.252374949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039050 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039070 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039106 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.03909174 +0000 UTC m=+46.252442444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : secret "router-metrics-certs-default" not found Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039126 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.039115447 +0000 UTC m=+46.252466131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:41.039221 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039186 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:41.039615 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.039233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.039220482 +0000 UTC m=+46.252571197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:41.052659 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.052633 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s"] Oct 29 22:14:41.068426 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.068403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:41.072573 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.071588 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs"] Oct 29 22:14:41.087948 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.087918 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6ldvb\"" Oct 29 22:14:41.088043 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.087998 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:41.088088 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.088054 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 22:14:41.090998 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.090978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:41.094132 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.094111 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s"] Oct 29 22:14:41.129066 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.128996 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs"] Oct 29 22:14:41.139824 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.139798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqr9\" (UniqueName: \"kubernetes.io/projected/e53f8e29-48b9-4952-b819-a2ccd0cc0d60-kube-api-access-fnqr9\") pod \"migrator-66df5d496b-b45zs\" (UID: \"e53f8e29-48b9-4952-b819-a2ccd0cc0d60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:41.139925 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.139865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh6wz\" (UniqueName: \"kubernetes.io/projected/ec88a159-0181-41a7-8d16-fbac28de5674-kube-api-access-bh6wz\") pod \"migrator-66df5d496b-ksr5s\" (UID: \"ec88a159-0181-41a7-8d16-fbac28de5674\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:41.139991 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.139926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:41.139991 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.139971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:41.140096 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.140049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:41.140271 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140254 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:41.140338 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140272 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6865b464d5-vpmdj: secret "image-registry-tls" not found Oct 29 22:14:41.140338 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140322 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls podName:a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.140306913 +0000 UTC m=+46.353657600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls") pod "image-registry-6865b464d5-vpmdj" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00") : secret "image-registry-tls" not found Oct 29 22:14:41.140450 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140433 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:14:41.140450 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140443 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d75cfc68-hctnk: secret "image-registry-tls" not found Oct 29 22:14:41.140562 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140474 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls podName:78c2f084-7af8-4fe9-824a-2e50a58eae25 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.140463439 +0000 UTC m=+46.353814125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls") pod "image-registry-84d75cfc68-hctnk" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25") : secret "image-registry-tls" not found Oct 29 22:14:41.140562 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140545 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:14:41.140663 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.140588 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls podName:edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.140575588 +0000 UTC m=+46.353926270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls") pod "dns-default-gxpdb" (UID: "edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3") : secret "dns-default-metrics-tls" not found Oct 29 22:14:41.144689 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.144666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:41.244332 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.240927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqr9\" (UniqueName: \"kubernetes.io/projected/e53f8e29-48b9-4952-b819-a2ccd0cc0d60-kube-api-access-fnqr9\") pod \"migrator-66df5d496b-b45zs\" (UID: \"e53f8e29-48b9-4952-b819-a2ccd0cc0d60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:41.244332 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.240989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh6wz\" (UniqueName: \"kubernetes.io/projected/ec88a159-0181-41a7-8d16-fbac28de5674-kube-api-access-bh6wz\") pod \"migrator-66df5d496b-ksr5s\" (UID: \"ec88a159-0181-41a7-8d16-fbac28de5674\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:41.244332 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.241049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:41.244332 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.241116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:41.244848 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.244822 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:41.245174 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.245135 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:14:41.245288 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.245272 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:43.245250318 +0000 UTC m=+40.458601002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:41.245369 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:41.245301 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:43.245289696 +0000 UTC m=+40.458640382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : configmap "kube-rbac-proxy" not found Oct 29 22:14:41.265336 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.265312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh6wz\" (UniqueName: \"kubernetes.io/projected/ec88a159-0181-41a7-8d16-fbac28de5674-kube-api-access-bh6wz\") pod \"migrator-66df5d496b-ksr5s\" (UID: \"ec88a159-0181-41a7-8d16-fbac28de5674\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:41.269732 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.269684 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqr9\" (UniqueName: \"kubernetes.io/projected/e53f8e29-48b9-4952-b819-a2ccd0cc0d60-kube-api-access-fnqr9\") pod \"migrator-66df5d496b-b45zs\" (UID: \"e53f8e29-48b9-4952-b819-a2ccd0cc0d60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:41.396143 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.396074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:41.421930 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.421908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:41.483052 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:41.482594 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aaca52_ea07_44d1_8f0f_acb055b79ea3.slice/crio-ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2 WatchSource:0}: Error finding container ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2: Status 404 returned error can't find the container with id ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2 Oct 29 22:14:41.485017 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.484913 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-25css"] Oct 29 22:14:41.724821 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.724782 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s"] Oct 29 22:14:41.730372 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:41.730343 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec88a159_0181_41a7_8d16_fbac28de5674.slice/crio-bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1 WatchSource:0}: Error finding container bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1: Status 404 returned error can't find the container with id bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1 Oct 29 22:14:41.752350 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.752320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-25css" event={"ID":"c2aaca52-ea07-44d1-8f0f-acb055b79ea3","Type":"ContainerStarted","Data":"52c8425be9c8ac52257f22d265c9ff5fd782b315d0e58b08b818476e5cdb2b23"} Oct 29 22:14:41.752736 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.752384 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-25css" event={"ID":"c2aaca52-ea07-44d1-8f0f-acb055b79ea3","Type":"ContainerStarted","Data":"ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2"} Oct 29 22:14:41.754378 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.754354 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" event={"ID":"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3","Type":"ContainerStarted","Data":"35c0f6006a7feb7f7dd0c07cee34c6757bececc1d46ad26f802705e62aab2e5f"} Oct 29 22:14:41.756139 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.756111 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" event={"ID":"ec88a159-0181-41a7-8d16-fbac28de5674","Type":"ContainerStarted","Data":"bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1"} Oct 29 22:14:41.761322 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.761270 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs"] Oct 29 22:14:41.770807 ip-10-0-134-94 kubenswrapper[2577]: W1029 22:14:41.770779 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53f8e29_48b9_4952_b819_a2ccd0cc0d60.slice/crio-7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5 WatchSource:0}: Error finding container 7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5: Status 404 returned error can't find the container with id 7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5 Oct 29 22:14:41.791235 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.791191 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-25css" podStartSLOduration=1.791175372 podStartE2EDuration="1.791175372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:41.790471916 +0000 UTC m=+39.003822643" watchObservedRunningTime="2026-10-29 22:14:41.791175372 +0000 UTC m=+39.004526078" Oct 29 22:14:41.832246 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:41.832199 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" podStartSLOduration=342.759239871 podStartE2EDuration="5m49.832189402s" totalImagesPullingTime="7.072949531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:41.831881945 +0000 UTC m=+39.045232649" watchObservedRunningTime="2026-10-29 22:14:41.832189402 +0000 UTC m=+39.045540105" Oct 29 22:14:42.764716 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:42.764609 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" event={"ID":"e53f8e29-48b9-4952-b819-a2ccd0cc0d60","Type":"ContainerStarted","Data":"7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5"} Oct 29 22:14:42.768904 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:42.768837 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k2gxs" event={"ID":"93780727-5cdc-4eaa-91fc-92e7c3889eaa","Type":"ContainerStarted","Data":"640431aa26416febce130720aa3cda00150de1204d928f261f5a9e5f35b90dd0"} Oct 29 22:14:42.769517 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:42.769313 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:42.807611 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:42.806363 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k2gxs" podStartSLOduration=33.273473953 podStartE2EDuration="38.80634587s" totalImagesPullingTime="5.532871917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:42.804520018 +0000 UTC m=+40.017870720" watchObservedRunningTime="2026-10-29 22:14:42.80634587 +0000 UTC m=+40.019696577" Oct 29 22:14:43.265950 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:43.265908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:43.266126 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:43.266001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:43.266126 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:43.266073 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:43.266126 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:43.266102 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:14:43.266321 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:43.266145 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.266125227 +0000 UTC m=+44.479475922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:43.266321 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:43.266182 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.266171534 +0000 UTC m=+44.479522216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : configmap "kube-rbac-proxy" not found Oct 29 22:14:43.611238 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:43.610477 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:43.611238 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:43.610547 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:43.611238 ip-10-0-134-94 kubenswrapper[2577]: I1029 22:14:43.610917 2577 scope.go:122] "RemoveContainer" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" Oct 29 22:14:43.611238 ip-10-0-134-94 kubenswrapper[2577]: E1029 22:14:43.611117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wsjxt_openshift-console-operator(01ed9874-d3cc-4b95-a198-348f68659495)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" Oct 29 22:14:44.621491 ip-10-0-134-94 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:14:44.643530 ip-10-0-134-94 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:14:44.643740 ip-10-0-134-94 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:14:44.643798 ip-10-0-134-94 systemd[1]: kubelet.service: Consumed 3.118s CPU time. Oct 29 22:14:44.653665 ip-10-0-134-94 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:14:44.740839 ip-10-0-134-94 kubenswrapper[5073]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:44.740839 ip-10-0-134-94 kubenswrapper[5073]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:14:44.740839 ip-10-0-134-94 kubenswrapper[5073]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:44.740839 ip-10-0-134-94 kubenswrapper[5073]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:44.742881 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742813 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.742881 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742881 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742885 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742888 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742890 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742893 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742896 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742898 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742902 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742905 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742907 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742910 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742913 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742916 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742919 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742923 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742928 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742931 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742933 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742936 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.742946 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742939 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742942 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742945 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742947 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742950 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742953 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742956 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742958 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742961 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742964 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742970 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742972 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742975 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742977 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742980 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742983 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742986 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742990 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742993 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742996 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.743436 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.742999 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743001 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743004 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743007 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743009 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743012 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743015 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743017 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743019 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743022 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743027 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743029 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743032 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743034 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743037 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743040 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743042 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743045 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743047 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743050 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.743911 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743053 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743055 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743058 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743060 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743063 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743065 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743068 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743070 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743073 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743075 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743078 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743080 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743083 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743085 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743088 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743090 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743093 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743095 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743097 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.744394 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743100 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743104 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743106 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743110 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743113 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743116 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743118 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743121 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743124 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743126 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743129 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743131 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743134 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743136 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743139 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743141 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743144 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743161 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743164 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.744856 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743166 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743169 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743175 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743177 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743180 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743182 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743185 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743187 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743944 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743951 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743954 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743958 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743961 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743963 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743966 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743970 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743973 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743976 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743978 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.745352 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743981 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743983 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743986 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743988 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743991 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743993 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743996 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.743998 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744001 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744004 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744007 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744009 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744012 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744016 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744020 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744023 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744025 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744028 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744030 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.745807 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744032 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744035 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744037 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744040 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744042 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744045 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744048 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744051 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744053 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744056 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744059 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744062 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744064 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744067 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744069 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744072 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744074 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744077 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744080 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744083 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.746322 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744086 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744088 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744091 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744094 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744096 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744099 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744103 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744107 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744111 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744114 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744117 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744120 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744123 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744126 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744129 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744131 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744134 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744137 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744140 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744142 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.746801 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744145 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744163 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744165 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744168 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744171 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744174 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744177 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744180 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744182 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744185 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744187 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744192 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744194 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744197 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744199 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744202 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744205 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744207 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744210 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744212 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.747295 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744214 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744217 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744220 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744222 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744225 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744227 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744230 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744233 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744236 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744239 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744241 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744244 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744246 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744249 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744251 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744254 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744731 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744737 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744740 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.747805 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744743 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744745 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744748 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744751 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744755 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744758 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744760 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744763 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744765 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744768 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744771 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744774 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744776 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744778 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744781 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744784 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744788 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744790 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744793 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744796 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.748268 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744798 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744801 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744803 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744806 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744808 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744811 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744813 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744816 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744818 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744821 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744824 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744827 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744829 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744832 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744834 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744837 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744840 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744843 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744845 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744848 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.748749 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744850 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744853 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744855 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744858 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744861 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744863 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744866 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744869 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744873 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744878 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744882 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744885 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744888 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744891 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744894 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744896 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744899 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744901 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744904 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.749242 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744906 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744909 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744911 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744918 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744921 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744923 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744927 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744930 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744934 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744938 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744941 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744944 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744947 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744950 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744952 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744966 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744969 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744972 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744974 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.749689 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744977 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744979 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744983 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744986 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744988 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744991 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744993 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744996 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.744998 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745001 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745003 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745006 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745008 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745010 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745013 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745015 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745018 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745021 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745023 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.750161 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745026 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745030 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745032 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745037 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745039 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745042 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745133 5073 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745143 5073 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745162 5073 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745167 5073 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745171 5073 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745174 5073 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745178 5073 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745182 5073 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745185 5073 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745190 5073 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745193 5073 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745196 5073 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745200 5073 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745203 5073 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745205 5073 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745209 5073 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745211 5073 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:14:44.750614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745214 5073 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745218 5073 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745220 5073 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745223 5073 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745226 5073 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745230 5073 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745233 5073 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745236 5073 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745240 5073 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745243 5073 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745246 5073 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745249 5073 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745253 5073 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745256 5073 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745259 5073 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745263 5073 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745266 5073 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745269 5073 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745271 5073 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745274 5073 flags.go:64] FLAG: --enable-server="true" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745277 5073 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745281 5073 flags.go:64] FLAG: --event-burst="100" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745285 5073 flags.go:64] FLAG: --event-qps="50" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745287 5073 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:14:44.751183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745292 5073 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745295 5073 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745299 5073 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745302 5073 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745305 5073 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745308 5073 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745311 5073 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745313 5073 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745316 5073 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745319 5073 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745322 5073 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745324 5073 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745327 5073 flags.go:64] FLAG: --feature-gates="" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745331 5073 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745334 5073 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745337 5073 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745341 5073 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745344 5073 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745348 5073 flags.go:64] FLAG: --help="false" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745350 5073 flags.go:64] FLAG: --hostname-override="ip-10-0-134-94.ec2.internal" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745355 5073 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745358 5073 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745361 5073 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745365 5073 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:14:44.751768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745368 5073 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745371 5073 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745374 5073 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745377 5073 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745380 5073 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745383 5073 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745386 5073 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745388 5073 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745393 5073 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745395 5073 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745398 5073 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745401 5073 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745404 5073 flags.go:64] FLAG: --lock-file="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745407 5073 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745410 5073 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745412 5073 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745417 5073 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745420 5073 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745423 5073 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745426 5073 flags.go:64] FLAG: --logging-format="text" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745429 5073 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745432 5073 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745435 5073 flags.go:64] FLAG: --manifest-url="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745438 5073 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745445 5073 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:14:44.752383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745450 5073 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745457 5073 flags.go:64] FLAG: --max-pods="110" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745461 5073 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745467 5073 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745471 5073 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745474 5073 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745477 5073 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745479 5073 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745482 5073 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745489 5073 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745492 5073 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745495 5073 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745498 5073 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745501 5073 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745503 5073 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745508 5073 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745511 5073 flags.go:64] FLAG: --port="10250" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745513 5073 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745516 5073 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-095a2cc92717d065a" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745519 5073 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745522 5073 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745525 5073 flags.go:64] FLAG: --register-node="true" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745528 5073 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745532 5073 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745534 5073 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745537 5073 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:14:44.752964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745540 5073 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745543 5073 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745546 5073 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745549 5073 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745552 5073 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745555 5073 flags.go:64] FLAG: --runonce="false" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745558 5073 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745561 5073 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745564 5073 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745568 5073 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745571 5073 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745574 5073 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745577 5073 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745580 5073 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745582 5073 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745585 5073 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745588 5073 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745591 5073 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745594 5073 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745597 5073 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745599 5073 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745606 5073 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745609 5073 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745612 5073 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745615 5073 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:14:44.753902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745618 5073 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745621 5073 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745624 5073 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745627 5073 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745630 5073 flags.go:64] FLAG: --v="2" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745634 5073 flags.go:64] FLAG: --version="false" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745638 5073 flags.go:64] FLAG: --vmodule="" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745641 5073 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.745644 5073 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745771 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745774 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745777 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745780 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745783 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745786 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745788 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745793 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745796 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745798 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745801 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745804 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.754767 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745806 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745809 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745811 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745814 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745817 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745819 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745822 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745826 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745828 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745831 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745833 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745836 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745838 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745841 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745843 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745846 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745849 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745851 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745853 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.755339 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745856 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745859 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745862 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745864 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745867 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745869 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745872 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745874 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745880 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745883 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745887 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745900 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745903 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745906 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745909 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745912 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745914 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745917 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745920 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.755790 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745922 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745927 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745929 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745932 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745934 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745937 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745939 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745942 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745944 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745947 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745949 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745952 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745955 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745957 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745961 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745965 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745967 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745970 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745973 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745975 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.756388 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745978 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745981 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745984 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745987 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745990 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745993 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745996 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.745998 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746001 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746003 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746005 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746008 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746010 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746014 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746017 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746019 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746022 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746024 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746027 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.756853 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746029 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746032 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746035 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746037 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746040 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746042 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746045 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746047 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746050 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746052 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746055 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746057 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746060 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746062 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746066 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746069 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.757335 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746071 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.746097 5073 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746131 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746136 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746139 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746142 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746145 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746164 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746169 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746172 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746177 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746180 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746182 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746185 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746189 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746192 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.757831 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746194 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746197 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746199 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746202 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746204 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746207 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746209 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746213 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746217 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746220 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746223 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746225 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746228 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746231 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746234 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746238 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746240 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746243 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746246 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746249 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.758255 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746251 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746254 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746256 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746258 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746261 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746263 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746268 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746270 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746273 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746275 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746279 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746283 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746286 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746289 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746291 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746293 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746296 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746299 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746301 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.758769 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746304 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746306 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746309 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746311 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746314 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746316 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746319 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746321 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746325 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746328 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746330 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746333 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746336 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746339 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746342 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746344 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746347 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746349 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746351 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.759226 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746355 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746357 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746360 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746362 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746364 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746367 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746369 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746372 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746374 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746377 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746379 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746382 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746384 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746387 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746389 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746391 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746394 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746396 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746399 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746401 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.759675 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746403 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746407 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746409 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746412 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746414 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746417 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746419 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746422 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746424 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746428 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746431 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746434 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746436 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.746439 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.749975 5073 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.749989 5073 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.750006 5073 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.750011 5073 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750065 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.760131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750070 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750073 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750076 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750079 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750083 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750088 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750090 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750094 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750097 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750099 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750102 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750104 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750107 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750110 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750112 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750115 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750117 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750120 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750123 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.760603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750126 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750129 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750132 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750134 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750137 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750140 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750142 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750145 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750165 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750168 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750171 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750174 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750177 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750179 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750182 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750185 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750187 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750190 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750192 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750195 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.761037 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750197 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750200 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750202 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750206 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750208 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750211 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750213 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750216 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750218 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750220 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750223 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750225 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750228 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750231 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750233 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750236 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750240 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750244 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750247 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.761512 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750250 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750252 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750255 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750258 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750260 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750262 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750265 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750267 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750270 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750272 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750275 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750277 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750279 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750282 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750284 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750287 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750289 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750292 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750294 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750297 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.761968 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750299 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750302 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750304 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750306 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750309 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750311 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750314 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750317 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750320 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750323 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750325 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750328 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750330 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750333 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750336 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750353 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750357 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750359 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750362 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.762442 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750365 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750368 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750370 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750373 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750375 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750378 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750380 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750382 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.750410 5073 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750537 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750542 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750545 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750548 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750550 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750553 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750555 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.762889 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750571 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750575 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750578 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750581 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750584 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750589 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750594 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750597 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750600 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750603 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750606 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750608 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750611 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750614 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750617 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750620 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750622 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750625 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750628 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.763286 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750631 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750633 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750636 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750638 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750641 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750643 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750646 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750648 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750653 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750656 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750658 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750661 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750663 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750666 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750668 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750671 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750674 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750676 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750679 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.763722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750682 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750684 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750687 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750689 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750692 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750694 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750697 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750699 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750702 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750705 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750707 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750709 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750712 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750715 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750717 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750719 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750722 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750725 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750727 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750730 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.764247 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750732 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750736 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750738 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750741 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750744 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750746 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750749 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750751 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750755 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750758 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750761 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750763 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750766 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750769 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750771 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750774 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750776 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750778 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750781 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.764733 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750783 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750786 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750788 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750790 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750793 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750796 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750799 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750801 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750804 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750807 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750809 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750811 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750814 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750816 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750820 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750822 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750825 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750827 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750830 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750833 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.765204 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750835 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750838 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.750860 5073 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750889 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750893 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750896 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750899 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750902 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750905 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750908 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750911 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750913 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750915 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750918 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750921 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750923 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:44.765684 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750925 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750928 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750930 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750933 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750935 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750938 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750940 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750943 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750945 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750949 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750952 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750957 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750959 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750964 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750966 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750969 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750971 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750974 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750976 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:44.766113 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750979 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750981 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750984 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750986 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750989 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750992 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750994 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750997 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.750999 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751002 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751005 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751008 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751011 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751013 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751015 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751018 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751020 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751023 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751025 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751028 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:44.766722 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751030 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751032 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751035 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751037 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751041 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751044 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751047 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751050 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751052 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751055 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751057 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751060 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751062 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751065 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751068 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751071 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751075 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751078 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751081 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751084 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:44.767213 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751086 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751089 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751091 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751094 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751096 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751099 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751101 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751104 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751107 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751109 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751111 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751114 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751116 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751119 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751121 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751125 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751129 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751131 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751135 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:44.767730 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751137 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751140 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751142 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751145 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751160 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751163 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751165 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751168 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751170 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751172 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751176 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751178 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751185 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751189 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:44.751192 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.751238 5073 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752179 5073 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752244 5073 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752428 5073 server.go:1014] "Starting client certificate rotation" Oct 29 22:14:44.768348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752567 5073 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:44.769090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752618 5073 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:09:03 +0000 UTC" deadline="2028-04-29 13:39:23.097230312 +0000 UTC" Oct 29 22:14:44.769090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752652 5073 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13143h24m38.344580389s" Oct 29 22:14:44.769090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.752905 5073 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:44.769090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.767765 5073 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:44.769458 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.769440 5073 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:14:44.771233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.771216 5073 log.go:25] "Validated CRI v1 image API" Oct 29 22:14:44.771906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.771890 5073 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:14:44.773985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.773960 5073 fs.go:135] Filesystem UUIDs: map[2d7c4d52-a99d-4ad1-b48d-e66e5a40fb59:/dev/nvme0n1p3 454241e5-f75e-4fbd-aadc-475895045f39:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 29 22:14:44.774248 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.773985 5073 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cda4a472c66dfdd7fb1a0fe509e5fd44f020e62cd8fe1e90a3cc9d621be4663/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cda4a472c66dfdd7fb1a0fe509e5fd44f020e62cd8fe1e90a3cc9d621be4663/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/879cec33bead00a540eaa87c45440fe3d155f754a197e880de4a32b7719dccb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/879cec33bead00a540eaa87c45440fe3d155f754a197e880de4a32b7719dccb6/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01ed9874-d3cc-4b95-a198-348f68659495/volumes/kubernetes.io~projected/kube-api-access-m8mwx:{mountpoint:/var/lib/kubelet/pods/01ed9874-d3cc-4b95-a198-348f68659495/volumes/kubernetes.io~projected/kube-api-access-m8mwx major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01ed9874-d3cc-4b95-a198-348f68659495/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/01ed9874-d3cc-4b95-a198-348f68659495/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volumes/kubernetes.io~projected/kube-api-access-vqtmn:{mountpoint:/var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volumes/kubernetes.io~projected/kube-api-access-vqtmn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~projected/kube-api-access-xwk7k:{mountpoint:/var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~projected/kube-api-access-xwk7k major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e74b42e-7da7-4a53-817b-5a8873b3377d/volumes/kubernetes.io~projected/kube-api-access-8pbqh:{mountpoint:/var/lib/kubelet/pods/2e74b42e-7da7-4a53-817b-5a8873b3377d/volumes/kubernetes.io~projected/kube-api-access-8pbqh major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e74b42e-7da7-4a53-817b-5a8873b3377d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2e74b42e-7da7-4a53-817b-5a8873b3377d/volumes/kubernetes.io~secret/cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/340bdb67-9b1d-40f0-8d58-db1157201e45/volumes/kubernetes.io~projected/kube-api-access-zkwrp:{mountpoint:/var/lib/kubelet/pods/340bdb67-9b1d-40f0-8d58-db1157201e45/volumes/kubernetes.io~projected/kube-api-access-zkwrp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c/volumes/kubernetes.io~projected/kube-api-access-5zbbl:{mountpoint:/var/lib/kubelet/pods/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c/volumes/kubernetes.io~projected/kube-api-access-5zbbl major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40badafa-2bba-47cc-b94c-c8f7233c7fb6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/40badafa-2bba-47cc-b94c-c8f7233c7fb6/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49aa972a-477a-419c-a2ba-9e5195039423/volumes/kubernetes.io~projected/kube-api-access-hx6vw:{mountpoint:/var/lib/kubelet/pods/49aa972a-477a-419c-a2ba-9e5195039423/volumes/kubernetes.io~projected/kube-api-access-hx6vw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d78fd0a-b2a6-4688-9435-1e5c972575f8/volumes/kubernetes.io~projected/kube-api-access-vk9vj:{mountpoint:/var/lib/kubelet/pods/4d78fd0a-b2a6-4688-9435-1e5c972575f8/volumes/kubernetes.io~projected/kube-api-access-vk9vj major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3/volumes/kubernetes.io~projected/kube-api-access-2wdjr:{mountpoint:/var/lib/kubelet/pods/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3/volumes/kubernetes.io~projected/kube-api-access-2wdjr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6c3496-906d-4d23-8e0c-a518b01f9ca3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5f6c3496-906d-4d23-8e0c-a518b01f9ca3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60db2db0-73c9-4cb7-aa1b-9c670677bc6e/volumes/kubernetes.io~projected/kube-api-access-68d5z:{mountpoint:/var/lib/kubelet/pods/60db2db0-73c9-4cb7-aa1b-9c670677bc6e/volumes/kubernetes.io~projected/kube-api-access-68d5z major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6813a426-c6a9-4a87-b2b4-8d60472da95d/volumes/kubernetes.io~projected/kube-api-access-n29lf:{mountpoint:/var/lib/kubelet/pods/6813a426-c6a9-4a87-b2b4-8d60472da95d/volumes/kubernetes.io~projected/kube-api-access-n29lf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~projected/kube-api-access-pcv66:{mountpoint:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~projected/kube-api-access-pcv66 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~projected/kube-api-access-vmljm:{mountpoint:/var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~projected/kube-api-access-vmljm major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c439b93-fb4b-441d-96d5-54852b26dcde/volumes/kubernetes.io~projected/kube-api-access-dtbdk:{mountpoint:/var/lib/kubelet/pods/7c439b93-fb4b-441d-96d5-54852b26dcde/volumes/kubernetes.io~projected/kube-api-access-dtbdk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93780727-5cdc-4eaa-91fc-92e7c3889eaa/volumes/kubernetes.io~projected/kube-api-access-xct6r:{mountpoint:/var/lib/kubelet/pods/93780727-5cdc-4eaa-91fc-92e7c3889eaa/volumes/kubernetes.io~projected/kube-api-access-xct6r major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1cc6e96-33dc-4727-b69a-c3050aee425c/volumes/kubernetes.io~projected/kube-api-access-4278f:{mountpoint:/var/lib/kubelet/pods/a1cc6e96-33dc-4727-b69a-c3050aee425c/volumes/kubernetes.io~projected/kube-api-access-4278f major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~projected/kube-api-access-lp4tm:{mountpoint:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~projected/kube-api-access-lp4tm major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c/volumes/kubernetes.io~projected/kube-api-access-fwdzn:{mountpoint:/var/lib/kubelet/pods/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c/volumes/kubernetes.io~projected/kube-api-access-fwdzn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9/volumes/kubernetes.io~projected/kube-api-access-vpfmp:{mountpoint:/var/lib/kubelet/pods/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9/volumes/kubernetes.io~projected/kube-api-access-vpfmp major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aaca52-ea07-44d1-8f0f-acb055b79ea3/volumes/kubernetes.io~projected/kube-api-access-bww45:{mountpoint:/var/lib/kubelet/pods/c2aaca52-ea07-44d1-8f0f-acb055b79ea3/volumes/kubernetes.io~projected/kube-api-access-bww45 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aaca52-ea07-44d1-8f0f-acb055b79ea3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c2aaca52-ea07-44d1-8f0f-acb055b79ea3/volumes/kubernetes.io~secret/signing-key major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2f1152f-ec5b-4fb7-8b17-04396337eb07/volumes/kubernetes.io~projected/kube-api-access-27kwm:{mountpoint:/var/lib/kubelet/pods/c2f1152f-ec5b-4fb7-8b17-04396337eb07/volumes/kubernetes.io~projected/kube-api-access-27kwm major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce594888-8263-4ccd-b24d-66900da5d257/volumes/kubernetes.io~projected/kube-api-access-ndm85:{mountpoint:/var/lib/kubelet/pods/ce594888-8263-4ccd-b24d-66900da5d257/volumes/kubernetes.io~projected/kube-api-access-ndm85 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce594888-8263-4ccd-b24d-66900da5d257/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ce594888-8263-4ccd-b24d-66900da5d257/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5414f15-670a-4ca7-a6c2-78a5426d717e/volumes/kubernetes.io~projected/kube-api-access-m79nh:{mountpoint:/var/lib/kubelet/pods/d5414f15-670a-4ca7-a6c2-78a5426d717e/volumes/kubernetes.io~projected/kube-api-access-m79nh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e53f8e29-48b9-4952-b819-a2ccd0cc0d60/volumes/kubernetes.io~projected/kube-api-access-fnqr9:{mountpoint:/var/lib/kubelet/pods/e53f8e29-48b9-4952-b819-a2ccd0cc0d60/volumes/kubernetes.io~projected/kube-api-access-fnqr9 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec88a159-0181-41a7-8d16-fbac28de5674/volumes/kubernetes.io~projected/kube-api-access-bh6wz:{mountpoint:/var/lib/kubelet/pods/ec88a159-0181-41a7-8d16-fbac28de5674/volumes/kubernetes.io~projected/kube-api-access-bh6wz major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3/volumes/kubernetes.io~projected/kube-api-access-t7rfk:{mountpoint:/var/lib/kubelet/pods/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3/volumes/kubernetes.io~projected/kube-api-access-t7rfk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f52c0b21-62cc-4ae5-b309-a79f19fa8974/volumes/kubernetes.io~projected/kube-api-access-wzhf5:{mountpoint:/var/lib/kubelet/pods/f52c0b21-62cc-4ae5-b309-a79f19fa8974/volumes/kubernetes.io~projected/kube-api-access-wzhf5 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f52c0b21-62cc-4ae5-b309-a79f19fa8974/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f52c0b21-62cc-4ae5-b309-a79f19fa8974/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7f1df3e-2020-4541-8b82-363ccdad9a05/volumes/kubernetes.io~projected/kube-api-access-p5gpd:{mountpoint:/var/lib/kubelet/pods/f7f1df3e-2020-4541-8b82-363ccdad9a05/volumes/kubernetes.io~projected/kube-api-access-p5gpd major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8e66c05-354b-4882-97a5-a3c247a548e8/volumes/kubernetes.io~projected/kube-api-access-pklpp:{mountpoint:/var/lib/kubelet/pods/f8e66c05-354b-4882-97a5-a3c247a548e8/volumes/kubernetes.io~projected/kube-api-access-pklpp major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8e66c05-354b-4882-97a5-a3c247a548e8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f8e66c05-354b-4882-97a5-a3c247a548e8/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbeceea4-8081-4c42-a085-07b5457ed5c7/volumes/kubernetes.io~projected/kube-api-access-b45sq:{mountpoint:/var/lib/kubelet/pods/fbeceea4-8081-4c42-a085-07b5457ed5c7/volumes/kubernetes.io~projected/kube-api-access-b45sq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd4aa13b-38c8-4806-978a-0ce17607e28b/volumes/kubernetes.io~projected/kube-api-access-69wgp:{mountpoint:/var/lib/kubelet/pods/fd4aa13b-38c8-4806-978a-0ce17607e28b/volumes/kubernetes.io~projected/kube-api-access-69wgp major:0 minor:321 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6cb426e7883c5a59cbc2547da1d29c6f0a10e045da9297e0fded0cd2baac3c86/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b75c25280506ca4c6256c1860d1cd83f2bfede21db0cf588e55af6efc48fff2c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/ff3de013edacff965ed97b5aec8dcd7fd51e0be4b9f0e71033bb441ee4a61c93/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/13636a7a8f7c0a38f23999d80ece58237785be9a4407113b5c6ca20719f7f53f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e301de3fea00973959c8ccf8ea156b34127604c9cdb46a39a7296faaf6de3068/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8204d31bd49324784ae9d8b3618c4d7c3a6b2bb36f7b9849e8344d9cde11bc11/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/53da71c0ee2349192697618f174fa757d2b16043c00eddd8b6384e26b565cfbb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0543524d200e1ba64c23207f2ce6a35fec1bd830addaa190902ba6b947893eb3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b94b17baac83972d7327e8528d10ed3fdbc1cf599477b1f157675ff89f6ce5df/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/af6c0149249286bd4d1ec938855082bc66a4461b60fcb4a749dac90931526de9/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/24455934a75d257127cd9014f8d1838f88b0599fbb41cedd9c7af6492e482781/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/2385943b4c321b5ff87451817b8cbdb648317f1d832c201d9a568d1816f27eac/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fdb4eba4762d58eec3693acd8bad4f3b4ea7a7cd050921462d696e331e0ad6c3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/38b1d3e983e83bcf081b750b4bec96d16809b5ffe9fe8c27333474ffff352341/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/de740f87f4426dcd07d752d605ab0cab0ebb6a5f1441aec89478fe49cbc8463c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0739b04903b57dafd91c8bbd4b7a40287d219bba9d74e663db5c1dd5aa9c9f9c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/38d2a3c9f3a9897d60cb0048390721bf0a5cc2f79e9b28c55ddb060b36188ed2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/3aff124fd8fe0d85381dd6650a7d29149998ed860d1bae638594c0649beb6511/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/39f599da339474cee88ce57cf63d1466cfc26a338a19ee9af4f3318b5e45b7d2/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0e6c1dbd31dc3c79251a7b6ed130cd149de2d0ab7300913c4f9e7f43107bd603/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d0d857814d4a6542e7ffee35125d9576ac59aa177c4a5a86f6ec4cd31df349b7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7b20ed6ef510dcb5213ee282b93507833eb0fa4a16848fea7d51a0a8d26ce276/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d98f80fd101f7d13bf1910fae6d06004fe15c8ce781c1553a05f5e6ea346f6b0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/909be3232ab12968b10d302fb1ce930231ab6e7058c543505494ebd942f2155b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/51be7e49d17884dd6310e9c354e03ea6f98dd8ef224bb40f65d10d6a6e7d6082/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/ca2a9d0d018a7cc13378682ebcd93ff26e90f270d23aa80900119a2c28c412be/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d61f89075f875607b14095eb9387dd1f75a5121d807dba1eec5036770ac4b1ff/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b426073454236ee4f39f71f4b2d2f068ead30ac7e78c4d8669bc80fea69ac7e3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/26500b22b12e4b1d3e4932df136489ce6c3a81a80ee9e2b67fc5d99b2af69690/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/270536f31a44575eb31422323508e14e2037bba64609034ba4fd69a3904bf1a3/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b96fdd90af26840c8dc7120ff70dd0232f098efd6a7d7c19564bc8f52586b215/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/60e63d13186b548518153519d7bf166f95bf819e41f06053c5c1d8725df01d74/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/ef71d3c2ca9d82a61e380d312317dd75da0a95fb535e931e0482239d419519fd/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/49964dcd767f05b4f4f6ca395f2f9840c097d08c82f63ba4cf4538bb3da69cbc/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/a8e927119631141a9495da466fcfbd2e6ab91c3c1c83fc851c77d8072f6ed08f/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/3da310427fe55002513e624a1e2eba32d5599e06a0bb7164e97e5f396fe4db82/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/8e2c870f4b9602722ea6d8708d006d44722fe1fc4ba1a17197decbb6dcda2418/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/2df949a8883395d3d25d3f1b2b43ea88dc0a87569fed5c289f03d31ad8535779/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/1466105959df692ad7fdddd7f7ac4f0aecd2a41470a1b39b4f2decb8fd757f84/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/b7dfeffcf75b03d133a9d60a654f02af745361ad68bd756fe3446164f73a7b35/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/0fd3cc3f727d7fa3ebe81a6f9c6fba9bbb58ecd88a9e728d2efdd76b93a30a94/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/aa6cefdef5ef3576cd546ae19d9f92cd840e499abb75b021841b19242f0d2091/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/70d01757dc1e99ab7ba20dca36a3c2e031e8263cb300a7e9f419ecae94ae6cb1/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/799210f72d013596343365b3c7815af12ea4444837c387f15736590f48a9e713/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/58a10fc9fe049c86a19b294901bf2da039d832386496f297f70e82c80360a86b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/4cf9427be3b3c5525d9482aebd2f83eb6fba1271d9771e0b90988dc06a0a9193/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/9c61a3666826a09edfcaee43f697112c9907775f7493ac54f7c4aa7ec8b7f99a/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/51c994b45e7499e159eefbef924139501a5595c29ad3d6780bb43f45771e428f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/1daed7185e74828b2fafec9ab703d1218014f4187070cab82dd7e15e2db3944f/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/a4d78595886be3b1dad4c081a55d6f94944e94851d44b2d6f0633986780288a0/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/347622a07e191cc99bff3c67d6842d895c6c373f1330e7607845efa042b94706/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/0dd59cb991aaf887d34d8b0810dd24d9ad8c7ff7618c796cbadb441635079b2e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/07e9262f2562dce3010a20631019bcdeeb1fa25c18fd8fe0af352b63eaf62992/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/40ae3447e21fba645d0ea8e6ad41199f298f3d544d0762437b6d4a6ba527d0ea/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5ad575d498a0244002cd13e5ac8fad87d3d0422fb6f7e71a0e28e87daf0dc732/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a23f9f29c9d7ca0ea5ca911890a5d7ebc6322cf517ea220a614e9a59d71de633/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b07f65c65cab043e3816e1b9412adf41300e112f69f4f672ee239c48c64e674a/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6970be99e34f2cc29277dde5f1c177fc145da9cc82b8f77657f6d2db8ae252af/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3bd08121c0cebfe1a78c243bfe22f097be54c7379544f462972068f6263f0f9a/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/138690528b967d6d5b025e7aa5446b624b3e4c81e220e6218be516bd3e225ada/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e62ba407f334be6d1ab29984b86e9944cb0c67cb4a7b56bac45f9aeedfc41f86/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/449b13fd582c3e699105578abc5f29a7665a1ae6c599db7ca0d26fc5e0dfd539/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 22:14:44.782463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.782075 5073 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:44.781582591 +0000 UTC m=+0.096976662 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3133257 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2537848478bde51310c38b6cece4aa SystemUUID:ec253784-8478-bde5-1310-c38b6cece4aa BootID:6e09bfe6-7c0e-464d-b358-fdcb2e66c23e Filesystems:[{Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6813a426-c6a9-4a87-b2b4-8d60472da95d/volumes/kubernetes.io~projected/kube-api-access-n29lf DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d5414f15-670a-4ca7-a6c2-78a5426d717e/volumes/kubernetes.io~projected/kube-api-access-m79nh DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7ca75a87e70a9df857e5ac3dd0b088eeb5a56f8546cb1dca30fb2532e4535f3/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13fae00e54ab01883adec35a12a484efe1bd59a4eeced2e64652f4740df7cbab/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~projected/kube-api-access-vmljm DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2249cfa1f9992ff990dc1a2ddfa14d6ef3357cb7a944b0886889fc299074f8a2/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01ed9874-d3cc-4b95-a198-348f68659495/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e53f8e29-48b9-4952-b819-a2ccd0cc0d60/volumes/kubernetes.io~projected/kube-api-access-fnqr9 DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce594888-8263-4ccd-b24d-66900da5d257/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc379abbf50d6ca0630f6c8a3c47c70410dd183cab5078d24777323ef9c5d1b1/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e74b42e-7da7-4a53-817b-5a8873b3377d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40a8355ec7ca73f864db14d171d88df90497463bc5be588c872916edd4e56c05/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~projected/kube-api-access-pcv66 DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0eb79d2e259be56cb7a93f744916e86719956254155ffa0f7f80f7fd91b34fd/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c/volumes/kubernetes.io~projected/kube-api-access-5zbbl DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e74b42e-7da7-4a53-817b-5a8873b3377d/volumes/kubernetes.io~projected/kube-api-access-8pbqh DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cda4a472c66dfdd7fb1a0fe509e5fd44f020e62cd8fe1e90a3cc9d621be4663/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aaca52-ea07-44d1-8f0f-acb055b79ea3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04d69ceca33101d3821c622e6115b52c93695afc88b935a7fa685fbe31770f5d/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f52c0b21-62cc-4ae5-b309-a79f19fa8974/volumes/kubernetes.io~projected/kube-api-access-wzhf5 DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7f1df3e-2020-4541-8b82-363ccdad9a05/volumes/kubernetes.io~projected/kube-api-access-p5gpd DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8e66c05-354b-4882-97a5-a3c247a548e8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3/volumes/kubernetes.io~projected/kube-api-access-2wdjr DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/60db2db0-73c9-4cb7-aa1b-9c670677bc6e/volumes/kubernetes.io~projected/kube-api-access-68d5z DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3/volumes/kubernetes.io~projected/kube-api-access-t7rfk DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01ed9874-d3cc-4b95-a198-348f68659495/volumes/kubernetes.io~projected/kube-api-access-m8mwx DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bd2957bdbaa833a2855d2324bc05d8208cc17ed3ac14136c7592db4740fb61b/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6c3496-906d-4d23-8e0c-a518b01f9ca3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49aa972a-477a-419c-a2ba-9e5195039423/volumes/kubernetes.io~projected/kube-api-access-hx6vw DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d78fd0a-b2a6-4688-9435-1e5c972575f8/volumes/kubernetes.io~projected/kube-api-access-vk9vj DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c49c5f9114276349cc91228624106a55da8613b2f6302c5c92233ee0cefaf7ec/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~projected/kube-api-access-xwk7k DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b39db4b13f3a161aa48444700289782f4cd6c36faf1d3868107ef869c2291a3c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40badafa-2bba-47cc-b94c-c8f7233c7fb6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/879cec33bead00a540eaa87c45440fe3d155f754a197e880de4a32b7719dccb6/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec88a159-0181-41a7-8d16-fbac28de5674/volumes/kubernetes.io~projected/kube-api-access-bh6wz DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f637c044e0b1008668480af8af80db1783c8a5fe61a20357138cbf72fdedd87c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f52c0b21-62cc-4ae5-b309-a79f19fa8974/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2f1152f-ec5b-4fb7-8b17-04396337eb07/volumes/kubernetes.io~projected/kube-api-access-27kwm DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7969a73ffa2ad5df2eb68a29f4efcf99a5c8e84a8d3347d58712e1c119449bc5/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9/volumes/kubernetes.io~projected/kube-api-access-vpfmp DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffcb4f0066bc19e7f087a9617653979c40d8ab1092ad7925d5d093c578b69a72/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8e66c05-354b-4882-97a5-a3c247a548e8/volumes/kubernetes.io~projected/kube-api-access-pklpp DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a1cc6e96-33dc-4727-b69a-c3050aee425c/volumes/kubernetes.io~projected/kube-api-access-4278f DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d44e7a2038d6b736cc6359532b84b1b84a218f7abec009292b830162699f495c/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/637974a3cf2146db1b91486ac109ae16054f889bcb6192fcc5b1253e110ded41/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecabf1a8fa3c51371288feb308a24d13f65f52bb57ef7b199844f921ab6e4dc2/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c/volumes/kubernetes.io~projected/kube-api-access-fwdzn DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/417652dafa0f76e95b6dc82520b56468e837078282e04f299f586222ba056b92/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce594888-8263-4ccd-b24d-66900da5d257/volumes/kubernetes.io~projected/kube-api-access-ndm85 DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b833e1af9e5e4ae041dff765748f7081555c5c67921c79674c71a077d68aaedc/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/340bdb67-9b1d-40f0-8d58-db1157201e45/volumes/kubernetes.io~projected/kube-api-access-zkwrp DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c76ef97e991fdf651a0c9b8f7f529c93574196b164e791bab1fd241a90570a53/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93780727-5cdc-4eaa-91fc-92e7c3889eaa/volumes/kubernetes.io~projected/kube-api-access-xct6r DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fd4aa13b-38c8-4806-978a-0ce17607e28b/volumes/kubernetes.io~projected/kube-api-access-69wgp DeviceMajor:0 DeviceMinor:321 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c439b93-fb4b-441d-96d5-54852b26dcde/volumes/kubernetes.io~projected/kube-api-access-dtbdk DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0762bfee-d0ac-4a88-bee5-212f3b230a2d/volumes/kubernetes.io~projected/kube-api-access-vqtmn DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes/kubernetes.io~projected/kube-api-access-lp4tm DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/181431d4d1b05ec936e54da6db5a1f6d4a5a2373fb4bb797633d1bfe1ac4f2c8/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e2f97c839e57fe5829d6bf1aaa285d916c33da94682bcfbfaaff2afa1600a52/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a66a3fc-f006-419b-9442-ef4d8da8bd5e/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbeceea4-8081-4c42-a085-07b5457ed5c7/volumes/kubernetes.io~projected/kube-api-access-b45sq DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aaca52-ea07-44d1-8f0f-acb055b79ea3/volumes/kubernetes.io~projected/kube-api-access-bww45 DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1829b88e967232c1b0d0b4123280f05d56e8e576bb3d421dc82ddd78b283eed2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04d69ceca33101d MacAddress:22:6e:f2:08:76:5c Speed:10000 Mtu:8901} {Name:181431d4d1b05ec MacAddress:06:90:31:58:7a:80 Speed:10000 Mtu:8901} {Name:2249cfa1f9992ff MacAddress:ce:af:77:3b:c9:62 Speed:10000 Mtu:8901} {Name:40a8355ec7ca73f MacAddress:4a:92:26:fd:0c:20 Speed:10000 Mtu:8901} {Name:4cda4a472c66dfd MacAddress:62:96:14:e4:dd:f3 Speed:10000 Mtu:8901} {Name:637974a3cf2146d MacAddress:ae:5b:6d:76:39:1d Speed:10000 Mtu:8901} {Name:6bd2957bdbaa833 MacAddress:4e:a8:c3:78:80:a7 Speed:10000 Mtu:8901} {Name:7969a73ffa2ad5d MacAddress:da:07:79:43:b0:b4 Speed:10000 Mtu:8901} {Name:879cec33bead00a MacAddress:4a:b9:7c:87:ef:97 Speed:10000 Mtu:8901} {Name:a0eb79d2e259be5 MacAddress:12:d9:af:3e:9d:66 Speed:10000 Mtu:8901} {Name:b833e1af9e5e4ae MacAddress:1e:db:21:82:56:75 Speed:10000 Mtu:8901} {Name:bc379abbf50d6ca MacAddress:2e:50:5e:18:e8:27 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:1a:85:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:10:b5:db:64:10 Speed:0 Mtu:8901} {Name:c76ef97e991fdf6 MacAddress:52:d7:22:b9:c0:bb Speed:10000 Mtu:8901} {Name:d44e7a2038d6b73 MacAddress:ae:b1:60:24:06:a5 Speed:10000 Mtu:8901} {Name:ecabf1a8fa3c513 MacAddress:aa:cc:50:19:75:32 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:1a:85:9f Speed:0 Mtu:9001} {Name:f7ca75a87e70a9d MacAddress:a2:ca:08:49:9e:5a Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:e8:09:49:8d:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:14:44.782463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.782467 5073 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:14:44.782721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.782508 5073 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:14:44.782856 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.782831 5073 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.782857 5073 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.783275 5073 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.783290 5073 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.783305 5073 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.783351 5073 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.783388 5073 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.783790 5073 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:14:44.784038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.784008 5073 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:14:44.784427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.784058 5073 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:14:44.784427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.784069 5073 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:14:44.784427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.784080 5073 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:14:44.784427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.784092 5073 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:14:44.784427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.784105 5073 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:14:44.785575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.785551 5073 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:14:44.785575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.785572 5073 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:44.786080 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.786063 5073 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:14:44.788196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788180 5073 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:14:44.788196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788195 5073 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:44.788292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788286 5073 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:14:44.788502 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788491 5073 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:14:44.788530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788520 5073 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:14:44.788619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788610 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788624 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788630 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788636 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788641 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788646 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788652 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:14:44.788656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788657 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:14:44.788865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788663 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:14:44.788865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788670 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:14:44.788865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788678 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:14:44.788865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788687 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:14:44.788865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788709 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:14:44.788865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.788715 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:14:44.791951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.791940 5073 server.go:1263] "Started kubelet" Oct 29 22:14:44.792020 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.791994 5073 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:14:44.792096 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.792050 5073 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:14:44.792142 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.792117 5073 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:14:44.792506 ip-10-0-134-94 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:14:44.792657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.792587 5073 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:14:44.793204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.793186 5073 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:14:44.798254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798214 5073 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:14:44.798516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798496 5073 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:44.798774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798698 5073 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:14:44.798774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798714 5073 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:14:44.798900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798823 5073 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:14:44.798964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798860 5073 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:04 +0000 UTC" deadline="2028-08-08 05:35:12.816464348 +0000 UTC" Oct 29 22:14:44.798964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.798944 5073 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15559h20m28.01752391s" Oct 29 22:14:44.799280 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:44.798757 5073 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-94.ec2.internal\" not found" Oct 29 22:14:44.800277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800034 5073 factory.go:55] Registering systemd factory Oct 29 22:14:44.800277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800055 5073 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:14:44.800588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800572 5073 factory.go:153] Registering CRI-O factory Oct 29 22:14:44.800716 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800594 5073 factory.go:223] Registration of the crio container factory successfully Oct 29 22:14:44.800716 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800656 5073 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:14:44.800716 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800681 5073 factory.go:103] Registering Raw factory Oct 29 22:14:44.800716 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.800697 5073 manager.go:1199] Started watching for new ooms in manager Oct 29 22:14:44.801083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.801064 5073 manager.go:319] Starting recovery of all containers Oct 29 22:14:44.804371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804325 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9" volumeName="kubernetes.io/projected/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-kube-api-access-vpfmp" seLinuxMountContext="" Oct 29 22:14:44.804371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804364 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ed9874-d3cc-4b95-a198-348f68659495" volumeName="kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-trusted-ca" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804380 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762bfee-d0ac-4a88-bee5-212f3b230a2d" volumeName="kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-config" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804394 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e8709f8-a9fb-4dbe-90ea-2be488f64dc0" volumeName="kubernetes.io/projected/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-kube-api-access-xwk7k" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804410 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40badafa-2bba-47cc-b94c-c8f7233c7fb6" volumeName="kubernetes.io/secret/40badafa-2bba-47cc-b94c-c8f7233c7fb6-agent-certs" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804423 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6813a426-c6a9-4a87-b2b4-8d60472da95d" volumeName="kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-cni-binary-copy" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804436 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804448 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804481 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804494 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e8709f8-a9fb-4dbe-90ea-2be488f64dc0" volumeName="kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-tuned" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804507 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340bdb67-9b1d-40f0-8d58-db1157201e45" volumeName="kubernetes.io/projected/340bdb67-9b1d-40f0-8d58-db1157201e45-kube-api-access-zkwrp" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804520 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6813a426-c6a9-4a87-b2b4-8d60472da95d" volumeName="kubernetes.io/projected/6813a426-c6a9-4a87-b2b4-8d60472da95d-kube-api-access-n29lf" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804532 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804544 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration" seLinuxMountContext="" Oct 29 22:14:44.804532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804560 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c439b93-fb4b-441d-96d5-54852b26dcde" volumeName="kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804573 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aaca52-ea07-44d1-8f0f-acb055b79ea3" volumeName="kubernetes.io/secret/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-key" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804599 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52c0b21-62cc-4ae5-b309-a79f19fa8974" volumeName="kubernetes.io/projected/f52c0b21-62cc-4ae5-b309-a79f19fa8974-kube-api-access-wzhf5" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804613 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a625d89c-ea34-4bfb-80d6-3bcbf5958f5c" volumeName="kubernetes.io/projected/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-kube-api-access-fwdzn" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804625 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340bdb67-9b1d-40f0-8d58-db1157201e45" volumeName="kubernetes.io/empty-dir/340bdb67-9b1d-40f0-8d58-db1157201e45-tmp-dir" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804637 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6c3496-906d-4d23-8e0c-a518b01f9ca3" volumeName="kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804648 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52c0b21-62cc-4ae5-b309-a79f19fa8974" volumeName="kubernetes.io/secret/f52c0b21-62cc-4ae5-b309-a79f19fa8974-serving-cert" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804660 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e66c05-354b-4882-97a5-a3c247a548e8" volumeName="kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-service-ca-bundle" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804672 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e66c05-354b-4882-97a5-a3c247a548e8" volumeName="kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-tmp" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804683 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e66c05-354b-4882-97a5-a3c247a548e8" volumeName="kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-snapshots" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804697 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1cc6e96-33dc-4727-b69a-c3050aee425c" volumeName="kubernetes.io/projected/a1cc6e96-33dc-4727-b69a-c3050aee425c-kube-api-access-4278f" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804709 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec88a159-0181-41a7-8d16-fbac28de5674" volumeName="kubernetes.io/projected/ec88a159-0181-41a7-8d16-fbac28de5674-kube-api-access-bh6wz" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804721 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3" volumeName="kubernetes.io/configmap/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-config-volume" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804733 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52c0b21-62cc-4ae5-b309-a79f19fa8974" volumeName="kubernetes.io/configmap/f52c0b21-62cc-4ae5-b309-a79f19fa8974-config" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804751 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f1df3e-2020-4541-8b82-363ccdad9a05" volumeName="kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804775 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e8709f8-a9fb-4dbe-90ea-2be488f64dc0" volumeName="kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-tmp" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804789 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804808 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e66c05-354b-4882-97a5-a3c247a548e8" volumeName="kubernetes.io/projected/f8e66c05-354b-4882-97a5-a3c247a548e8-kube-api-access-pklpp" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804821 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeceea4-8081-4c42-a085-07b5457ed5c7" volumeName="kubernetes.io/configmap/fbeceea4-8081-4c42-a085-07b5457ed5c7-serviceca" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804834 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd4aa13b-38c8-4806-978a-0ce17607e28b" volumeName="kubernetes.io/empty-dir/fd4aa13b-38c8-4806-978a-0ce17607e28b-data-volume" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804849 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ed9874-d3cc-4b95-a198-348f68659495" volumeName="kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-config" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804862 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762bfee-d0ac-4a88-bee5-212f3b230a2d" volumeName="kubernetes.io/projected/0762bfee-d0ac-4a88-bee5-212f3b230a2d-kube-api-access-vqtmn" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804875 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a66a3fc-f006-419b-9442-ef4d8da8bd5e" volumeName="kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-default-certificate" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804887 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c439b93-fb4b-441d-96d5-54852b26dcde" volumeName="kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804900 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f1df3e-2020-4541-8b82-363ccdad9a05" volumeName="kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804913 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e66c05-354b-4882-97a5-a3c247a548e8" volumeName="kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804936 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762bfee-d0ac-4a88-bee5-212f3b230a2d" volumeName="kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-env-overrides" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804949 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c2f084-7af8-4fe9-824a-2e50a58eae25" volumeName="kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804962 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c439b93-fb4b-441d-96d5-54852b26dcde" volumeName="kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-binary-copy" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804973 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f1df3e-2020-4541-8b82-363ccdad9a05" volumeName="kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804986 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762bfee-d0ac-4a88-bee5-212f3b230a2d" volumeName="kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.804998 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a625d89c-ea34-4bfb-80d6-3bcbf5958f5c" volumeName="kubernetes.io/configmap/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805013 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce594888-8263-4ccd-b24d-66900da5d257" volumeName="kubernetes.io/secret/ce594888-8263-4ccd-b24d-66900da5d257-serving-cert" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805024 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ed9874-d3cc-4b95-a198-348f68659495" volumeName="kubernetes.io/projected/01ed9874-d3cc-4b95-a198-348f68659495-kube-api-access-m8mwx" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805037 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ed9874-d3cc-4b95-a198-348f68659495" volumeName="kubernetes.io/secret/01ed9874-d3cc-4b95-a198-348f68659495-serving-cert" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805050 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e74b42e-7da7-4a53-817b-5a8873b3377d" volumeName="kubernetes.io/secret/2e74b42e-7da7-4a53-817b-5a8873b3377d-cert" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805063 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60db2db0-73c9-4cb7-aa1b-9c670677bc6e" volumeName="kubernetes.io/projected/60db2db0-73c9-4cb7-aa1b-9c670677bc6e-kube-api-access-68d5z" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805078 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805091 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805104 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets" seLinuxMountContext="" Oct 29 22:14:44.805125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805116 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce594888-8263-4ccd-b24d-66900da5d257" volumeName="kubernetes.io/projected/ce594888-8263-4ccd-b24d-66900da5d257-kube-api-access-ndm85" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805171 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be7200fc-4e5d-416b-8898-2a8bac5648aa" volumeName="kubernetes.io/configmap/be7200fc-4e5d-416b-8898-2a8bac5648aa-nginx-conf" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805197 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aaca52-ea07-44d1-8f0f-acb055b79ea3" volumeName="kubernetes.io/configmap/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-cabundle" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805212 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762bfee-d0ac-4a88-bee5-212f3b230a2d" volumeName="kubernetes.io/secret/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805226 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e74b42e-7da7-4a53-817b-5a8873b3377d" volumeName="kubernetes.io/projected/2e74b42e-7da7-4a53-817b-5a8873b3377d-kube-api-access-8pbqh" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805239 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c439b93-fb4b-441d-96d5-54852b26dcde" volumeName="kubernetes.io/projected/7c439b93-fb4b-441d-96d5-54852b26dcde-kube-api-access-dtbdk" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805251 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e53f8e29-48b9-4952-b819-a2ccd0cc0d60" volumeName="kubernetes.io/projected/e53f8e29-48b9-4952-b819-a2ccd0cc0d60-kube-api-access-fnqr9" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805264 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3" volumeName="kubernetes.io/empty-dir/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-tmp-dir" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805277 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e66c05-354b-4882-97a5-a3c247a548e8" volumeName="kubernetes.io/secret/f8e66c05-354b-4882-97a5-a3c247a548e8-serving-cert" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805289 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49aa972a-477a-419c-a2ba-9e5195039423" volumeName="kubernetes.io/projected/49aa972a-477a-419c-a2ba-9e5195039423-kube-api-access-hx6vw" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805303 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d78fd0a-b2a6-4688-9435-1e5c972575f8" volumeName="kubernetes.io/projected/4d78fd0a-b2a6-4688-9435-1e5c972575f8-kube-api-access-vk9vj" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805315 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805327 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805342 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2f1152f-ec5b-4fb7-8b17-04396337eb07" volumeName="kubernetes.io/configmap/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-config" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805354 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5414f15-670a-4ca7-a6c2-78a5426d717e" volumeName="kubernetes.io/projected/d5414f15-670a-4ca7-a6c2-78a5426d717e-kube-api-access-m79nh" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805367 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93780727-5cdc-4eaa-91fc-92e7c3889eaa" volumeName="kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805380 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aaca52-ea07-44d1-8f0f-acb055b79ea3" volumeName="kubernetes.io/projected/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-kube-api-access-bww45" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805394 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce594888-8263-4ccd-b24d-66900da5d257" volumeName="kubernetes.io/configmap/ce594888-8263-4ccd-b24d-66900da5d257-config" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805406 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2f1152f-ec5b-4fb7-8b17-04396337eb07" volumeName="kubernetes.io/projected/c2f1152f-ec5b-4fb7-8b17-04396337eb07-kube-api-access-27kwm" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805419 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f1df3e-2020-4541-8b82-363ccdad9a05" volumeName="kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805432 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805444 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3" volumeName="kubernetes.io/projected/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-kube-api-access-t7rfk" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805458 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeceea4-8081-4c42-a085-07b5457ed5c7" volumeName="kubernetes.io/projected/fbeceea4-8081-4c42-a085-07b5457ed5c7-kube-api-access-b45sq" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805472 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd4aa13b-38c8-4806-978a-0ce17607e28b" volumeName="kubernetes.io/projected/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-api-access-69wgp" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805485 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40badafa-2bba-47cc-b94c-c8f7233c7fb6" volumeName="kubernetes.io/configmap/40badafa-2bba-47cc-b94c-c8f7233c7fb6-konnectivity-ca" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805497 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547cbb0b-f75f-4a5d-80ee-4d7073cc76f3" volumeName="kubernetes.io/projected/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3-kube-api-access-2wdjr" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805521 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6813a426-c6a9-4a87-b2b4-8d60472da95d" volumeName="kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-daemon-config" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805533 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a66a3fc-f006-419b-9442-ef4d8da8bd5e" volumeName="kubernetes.io/projected/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-kube-api-access-vmljm" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805545 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9" volumeName="kubernetes.io/configmap/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-telemetry-config" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805557 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c" volumeName="kubernetes.io/projected/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c-kube-api-access-5zbbl" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805567 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a66a3fc-f006-419b-9442-ef4d8da8bd5e" volumeName="kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-stats-auth" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805578 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" volumeName="kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted" seLinuxMountContext="" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805590 5073 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:14:44.806966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.805598 5073 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:14:44.819178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.819136 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/1.log" Oct 29 22:14:44.821697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.821671 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/1.log" Oct 29 22:14:44.828072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.827671 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:44.828726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.828567 5073 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:14:44.829279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.829114 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:44.830455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.830179 5073 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:14:44.830455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.830199 5073 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:14:44.830455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.830218 5073 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:14:44.830455 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:44.830257 5073 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:14:44.838453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.838435 5073 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:44.841754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.841712 5073 generic.go:356] "Generic (PLEG): container finished" podID="daccf67d50dfb71546a9fa0941d51e11" containerID="88b68f8819125574d4bb62968276b1bfcf0a9e5ebf383c4df331f22fc8cfa085" exitCode=0 Oct 29 22:14:44.851539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.851508 5073 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:44.851740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.851724 5073 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:14:44.859870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.859847 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:14:44.860237 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.860215 5073 generic.go:356] "Generic (PLEG): container finished" podID="0762bfee-d0ac-4a88-bee5-212f3b230a2d" containerID="b65fca3491b518a929662565654683a53c108068414744c749ce694a9bcc69a6" exitCode=1 Oct 29 22:14:44.863500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.863479 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/1.log" Oct 29 22:14:44.863567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.863511 5073 generic.go:356] "Generic (PLEG): container finished" podID="01ed9874-d3cc-4b95-a198-348f68659495" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" exitCode=255 Oct 29 22:14:44.864582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.864560 5073 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:44.864751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.864733 5073 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:44.876435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.876381 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="69f7676af8ab1b0bace5871bca76c3bc3d6090dd16db75bd84bf18922b2d3681" exitCode=0 Oct 29 22:14:44.876435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.876403 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="22fc1d9d0ae6fddd30897fbe9c9714b1d185cc2ba850743c4016b9decda30280" exitCode=0 Oct 29 22:14:44.876435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.876414 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="08ba8bef42f88f015121d06c5b25b0def6720e3781b961a1922e67b17a2fb723" exitCode=0 Oct 29 22:14:44.876435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.876422 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="0406a95a779a6432af47d8ddd9a28a4a656f5d8c437e169524e8b5230df8760e" exitCode=0 Oct 29 22:14:44.876435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.876430 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="98529ed38296839528aaa4c7f492896b6a3ebadaae855e51f62e6f2cc6c15d58" exitCode=0 Oct 29 22:14:44.876698 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.876437 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c439b93-fb4b-441d-96d5-54852b26dcde" containerID="1aa8b72d320aa24b7abc7e2c3b8fd7ebd9b1d35de25b3c26a6efa8381b68c91b" exitCode=0 Oct 29 22:14:44.903601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.903584 5073 manager.go:324] Recovery completed Oct 29 22:14:44.923403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923360 5073 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:14:44.923403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923401 5073 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:14:44.923548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923426 5073 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:14:44.923629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923609 5073 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:14:44.923675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923631 5073 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:14:44.923675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923653 5073 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:14:44.923675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923661 5073 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:14:44.923675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923669 5073 policy_none.go:50] "Start" Oct 29 22:14:44.923842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923688 5073 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:14:44.923842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923701 5073 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:14:44.923931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923920 5073 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:14:44.923974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923930 5073 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:14:44.923974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.923939 5073 policy_none.go:44] "Start" Oct 29 22:14:44.928700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.928675 5073 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:14:44.930325 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:44.930304 5073 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 29 22:14:44.932113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.932102 5073 manager.go:341] "Starting Device Plugin manager" Oct 29 22:14:44.932615 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:44.932597 5073 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:14:44.932615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.932614 5073 server.go:87] "Starting device plugin registration server" Oct 29 22:14:44.932864 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.932852 5073 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:14:44.932896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.932867 5073 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:14:44.932997 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.932964 5073 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:14:44.933124 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.933085 5073 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:14:44.933214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.933133 5073 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:14:44.933214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.933141 5073 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:14:44.933319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.933242 5073 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:44.933121716Z","UUID":"11e34f0e-a156-4969-a534-365435c4b885","Handler":null,"Name":"","Endpoint":""} Oct 29 22:14:44.934467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.934449 5073 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:14:44.934539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:44.934476 5073 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:14:44.934883 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:44.934862 5073 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:14:45.033076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.033054 5073 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:45.034305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.034288 5073 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:45.034391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.034315 5073 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:45.034391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.034325 5073 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-94.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:45.034391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.034361 5073 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.070512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.070496 5073 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.070588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.070548 5073 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.131061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131001 5073 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal"] Oct 29 22:14:45.131571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131545 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" event={"ID":"daccf67d50dfb71546a9fa0941d51e11","Type":"ContainerStarted","Data":"aed67bf1a7b2a60f11f7d2ef9b644806206c2c33c4b80e0a120a23774a82c0e5"} Oct 29 22:14:45.131616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131579 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" event={"ID":"daccf67d50dfb71546a9fa0941d51e11","Type":"ContainerDied","Data":"88b68f8819125574d4bb62968276b1bfcf0a9e5ebf383c4df331f22fc8cfa085"} Oct 29 22:14:45.131616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131593 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" event={"ID":"daccf67d50dfb71546a9fa0941d51e11","Type":"ContainerStarted","Data":"3cff76dcdd9e83d8019d4b9a6f8f40594673aada3c1ba9e5d0a8ab0cd1c11187"} Oct 29 22:14:45.131616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131610 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.131766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131735 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" event={"ID":"d6ac7ba86bb940ac627127f0b9857b88","Type":"ContainerStarted","Data":"26846ecbd2ad1f1e773d40d4f79a94adab765e12584e2b2f24d482a6294fef70"} Oct 29 22:14:45.131766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131748 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" event={"ID":"d6ac7ba86bb940ac627127f0b9857b88","Type":"ContainerStarted","Data":"fbee8c42e96537d9cc96640069a9bf3eabfff4ab945bdb0f7c078bacb36a88c1"} Oct 29 22:14:45.131868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.131843 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.153827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.153809 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:45.153899 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:45.153841 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.154315 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.154299 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:45.154408 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:45.154330 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.254188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.254166 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.254269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.254198 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.254269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.254223 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6ac7ba86bb940ac627127f0b9857b88-config\") pod \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" (UID: \"d6ac7ba86bb940ac627127f0b9857b88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.354663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.354640 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.354758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.354684 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.354758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.354701 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6ac7ba86bb940ac627127f0b9857b88-config\") pod \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" (UID: \"d6ac7ba86bb940ac627127f0b9857b88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.354758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.354648 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.354758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.354733 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6ac7ba86bb940ac627127f0b9857b88-config\") pod \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" (UID: \"d6ac7ba86bb940ac627127f0b9857b88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.354901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.354790 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daccf67d50dfb71546a9fa0941d51e11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" (UID: \"daccf67d50dfb71546a9fa0941d51e11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.785243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.785220 5073 apiserver.go:52] "Watching apiserver" Oct 29 22:14:45.841313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.841288 5073 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:14:45.842068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.842040 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx","openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4","kube-system/kas-connection-checker-685895c7c9-vhdkj","kube-system/global-pull-secret-syncer-px4td","openshift-image-registry/image-registry-6865b464d5-vpmdj","openshift-service-ca/service-ca-5fffc97bdf-25css","openshift-dns/node-resolver-tbx55","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal","openshift-network-operator/iptables-alerter-lc8wg","openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s","kube-system/kas-connection-checker-685895c7c9-5p8jz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d","kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal","openshift-multus/network-metrics-daemon-8j2s2","openshift-ovn-kubernetes/ovnkube-node-hx452","openshift-ingress/router-default-b64c8c7d5-7zxkb","openshift-image-registry/image-registry-84d75cfc68-hctnk","openshift-image-registry/node-ca-2cvqq","openshift-multus/multus-7qnrm","openshift-insights/insights-operator-7c4986d449-t6pf2","openshift-network-console/networking-console-plugin-86c478bb88-pcdq8","openshift-insights/insights-runtime-extractor-p6ccn","openshift-ingress-canary/ingress-canary-97h69","openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br","kube-system/konnectivity-agent-mmmgd","openshift-cluster-node-tuning-operator/tuned-5jrfz","openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g","openshift-network-diagnostics/network-check-target-k2gxs","openshift-multus/multus-additional-cni-plugins-t7vcg","openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc","kube-system/kas-connection-checker-685895c7c9-6vb55","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk","openshift-console-operator/console-operator-7cd56b8dbc-wsjxt","openshift-dns/dns-default-gxpdb"] Oct 29 22:14:45.843573 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.843549 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.843896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.843868 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.844641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.844624 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:45.845121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.845104 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:45.847306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.847291 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:45.848994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.848976 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:45.856492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.856470 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.857110 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.857091 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.858708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.858691 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.859867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.859849 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.859867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.859866 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.860006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.859914 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.860389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860366 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 22:14:45.860483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860413 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 22:14:45.860483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860418 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 22:14:45.860793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860774 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 22:14:45.860793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860789 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 22:14:45.860911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860774 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.860911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860838 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 22:14:45.860911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860842 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.860911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860848 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 22:14:45.860911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860852 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 22:14:45.861181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860922 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.861181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860964 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.861181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860968 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.861181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.860964 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:14:45.861412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.861392 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 22:14:45.861870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.861837 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 22:14:45.869949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.869934 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:14:45.870055 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.870043 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 22:14:45.870097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.870043 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 22:14:45.870097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.870062 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 22:14:45.872648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872619 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2cw6s\"" Oct 29 22:14:45.872728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872651 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 22:14:45.872728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872664 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w24c4\"" Oct 29 22:14:45.872728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872688 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 22:14:45.872878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872859 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:14:45.872930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872892 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm48d\"" Oct 29 22:14:45.872930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872906 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:14:45.873014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.872977 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hznbv\"" Oct 29 22:14:45.873056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.873036 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxx6w\"" Oct 29 22:14:45.873699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.873683 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-475gp\"" Oct 29 22:14:45.876983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.876963 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 22:14:45.877392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877357 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pdxn8\"" Oct 29 22:14:45.877392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877362 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:14:45.877392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877387 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:14:45.877541 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877394 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbljw\"" Oct 29 22:14:45.877541 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877368 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:14:45.877798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877778 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qnnrl\"" Oct 29 22:14:45.877798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877807 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6ldvb\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877825 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877824 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kk6hd\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877842 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877845 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fw4tl\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877866 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877879 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-558gk\"" Oct 29 22:14:45.877971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.877940 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkqw\"" Oct 29 22:14:45.878244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.878000 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:14:45.878244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.878046 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9x6m\"" Oct 29 22:14:45.878244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.878105 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h6g72\"" Oct 29 22:14:45.878244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.878145 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 22:14:45.878244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.878188 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:14:45.887173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887138 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.887259 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887189 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.887342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887328 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.887342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887335 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.887493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887356 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 22:14:45.887493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887374 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.887493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887374 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 22:14:45.887493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887386 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 22:14:45.887493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887399 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.887493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887423 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.887799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887529 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.887799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887608 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:14:45.887799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887617 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:14:45.887799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887704 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 22:14:45.887799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887717 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887855 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887861 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887860 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887880 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887902 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887948 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.887964 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888047 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888072 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888090 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888099 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888225 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888227 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888253 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888265 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888287 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888308 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888317 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888349 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888362 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888369 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888384 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888388 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888397 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888352 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888353 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888368 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888411 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888410 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888443 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888455 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888464 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888500 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:14:45.888922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.888880 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:14:45.889305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.889293 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 22:14:45.891327 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.890580 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 22:14:45.892984 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.892958 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 22:14:45.895073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.895058 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.895234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.895219 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.900694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.900678 5073 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:14:45.915632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.915613 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:45.915698 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:45.915650 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-94.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.916185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.916169 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:45.916260 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:45.916204 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-94.ec2.internal" Oct 29 22:14:45.956514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-multus-certs\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.956600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956521 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.956600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956537 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.956600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956557 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-node-log\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.956759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956628 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-stats-auth\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.956759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956681 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:45.956869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956759 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.956869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956801 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdjr\" (UniqueName: \"kubernetes.io/projected/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3-kube-api-access-2wdjr\") pod \"network-check-source-6f78d59f99-9xtxc\" (UID: \"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:45.956869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956833 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbeceea4-8081-4c42-a085-07b5457ed5c7-serviceca\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:45.956869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956851 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9vj\" (UniqueName: \"kubernetes.io/projected/4d78fd0a-b2a6-4688-9435-1e5c972575f8-kube-api-access-vk9vj\") pod \"kas-connection-checker-685895c7c9-5p8jz\" (UID: \"4d78fd0a-b2a6-4688-9435-1e5c972575f8\") " pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:45.956869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956854 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:45.956869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956867 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-log-socket\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956891 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-default-certificate\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956898 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-stats-auth\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956911 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmljm\" (UniqueName: \"kubernetes.io/projected/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-kube-api-access-vmljm\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956954 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-config\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.956987 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-dbus\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957012 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-hostroot\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.957136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957123 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957185 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-default-certificate\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957193 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fbeceea4-8081-4c42-a085-07b5457ed5c7-serviceca\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957220 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957258 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f52c0b21-62cc-4ae5-b309-a79f19fa8974-serving-cert\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957276 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-config\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957305 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-cnibin\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957337 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957364 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-config-volume\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957392 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e66c05-354b-4882-97a5-a3c247a548e8-serving-cert\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.957420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957410 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklpp\" (UniqueName: \"kubernetes.io/projected/f8e66c05-354b-4882-97a5-a3c247a548e8-kube-api-access-pklpp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957512 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-bin\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957566 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce594888-8263-4ccd-b24d-66900da5d257-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957575 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-config-volume\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957619 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e66c05-354b-4882-97a5-a3c247a548e8-serving-cert\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957637 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-tmp\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957666 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-kubelet-config\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957623 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f52c0b21-62cc-4ae5-b309-a79f19fa8974-serving-cert\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957688 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd4aa13b-38c8-4806-978a-0ce17607e28b-crio-socket\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957714 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce594888-8263-4ccd-b24d-66900da5d257-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957777 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-tmp\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.957830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957819 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-etc-selinux\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957877 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce594888-8263-4ccd-b24d-66900da5d257-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957923 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bdb67-9b1d-40f0-8d58-db1157201e45-tmp-dir\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957948 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957973 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rfk\" (UniqueName: \"kubernetes.io/projected/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-kube-api-access-t7rfk\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957985 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce594888-8263-4ccd-b24d-66900da5d257-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957994 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bdb67-9b1d-40f0-8d58-db1157201e45-tmp-dir\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.957997 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958036 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-slash\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958077 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958129 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-ovn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958166 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-env-overrides\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958186 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958203 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-registration-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958230 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958266 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-cnibin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958295 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-key\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958314 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-env-overrides\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.958320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958319 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958385 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-service-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958457 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-netns\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958472 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-key\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958476 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtmn\" (UniqueName: \"kubernetes.io/projected/0762bfee-d0ac-4a88-bee5-212f3b230a2d-kube-api-access-vqtmn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958509 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958584 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kwm\" (UniqueName: \"kubernetes.io/projected/c2f1152f-ec5b-4fb7-8b17-04396337eb07-kube-api-access-27kwm\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958611 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4278f\" (UniqueName: \"kubernetes.io/projected/a1cc6e96-33dc-4727-b69a-c3050aee425c-kube-api-access-4278f\") pod \"volume-data-source-validator-58c9f5f46d-s4rsk\" (UID: \"a1cc6e96-33dc-4727-b69a-c3050aee425c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958618 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-service-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958634 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be7200fc-4e5d-416b-8898-2a8bac5648aa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958658 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29lf\" (UniqueName: \"kubernetes.io/projected/6813a426-c6a9-4a87-b2b4-8d60472da95d-kube-api-access-n29lf\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958682 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958708 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdzn\" (UniqueName: \"kubernetes.io/projected/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-kube-api-access-fwdzn\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958732 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40badafa-2bba-47cc-b94c-c8f7233c7fb6-agent-certs\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958755 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340bdb67-9b1d-40f0-8d58-db1157201e45-hosts-file\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958782 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-cni-binary-copy\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958806 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-kubelet\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958827 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be7200fc-4e5d-416b-8898-2a8bac5648aa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958830 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958893 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958962 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.958986 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbdk\" (UniqueName: \"kubernetes.io/projected/7c439b93-fb4b-441d-96d5-54852b26dcde-kube-api-access-dtbdk\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959005 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959030 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfmp\" (UniqueName: \"kubernetes.io/projected/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-kube-api-access-vpfmp\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959076 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-cni-binary-copy\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959122 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40badafa-2bba-47cc-b94c-c8f7233c7fb6-agent-certs\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:45.959178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959140 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959199 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-conf-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959224 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959270 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-host-slash\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959294 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh6wz\" (UniqueName: \"kubernetes.io/projected/ec88a159-0181-41a7-8d16-fbac28de5674-kube-api-access-bh6wz\") pod \"migrator-66df5d496b-ksr5s\" (UID: \"ec88a159-0181-41a7-8d16-fbac28de5674\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959321 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-var-lib-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959341 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79nh\" (UniqueName: \"kubernetes.io/projected/d5414f15-670a-4ca7-a6c2-78a5426d717e-kube-api-access-m79nh\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959365 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e74b42e-7da7-4a53-817b-5a8873b3377d-cert\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959381 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-system-cni-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959475 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959542 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-snapshots\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959607 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-systemd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959637 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbqh\" (UniqueName: \"kubernetes.io/projected/2e74b42e-7da7-4a53-817b-5a8873b3377d-kube-api-access-8pbqh\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959648 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e74b42e-7da7-4a53-817b-5a8873b3377d-cert\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959692 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-snapshots\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959772 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959778 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959813 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959839 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959865 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-config\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.959904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959898 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndm85\" (UniqueName: \"kubernetes.io/projected/ce594888-8263-4ccd-b24d-66900da5d257-kube-api-access-ndm85\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959919 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-device-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959942 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68d5z\" (UniqueName: \"kubernetes.io/projected/60db2db0-73c9-4cb7-aa1b-9c670677bc6e-kube-api-access-68d5z\") pod \"kas-connection-checker-685895c7c9-vhdkj\" (UID: \"60db2db0-73c9-4cb7-aa1b-9c670677bc6e\") " pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959963 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.959983 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960028 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-config\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960032 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww45\" (UniqueName: \"kubernetes.io/projected/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-kube-api-access-bww45\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960062 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960069 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-script-lib\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960110 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960115 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6vw\" (UniqueName: \"kubernetes.io/projected/49aa972a-477a-419c-a2ba-9e5195039423-kube-api-access-hx6vw\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960171 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960288 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960294 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovnkube-script-lib\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960324 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-original-pull-secret\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960325 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-bin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960355 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-multus\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960384 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-sys-fs\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960411 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysconfig\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960441 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-sys\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960457 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960472 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-system-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960494 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960520 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mwx\" (UniqueName: \"kubernetes.io/projected/01ed9874-d3cc-4b95-a198-348f68659495-kube-api-access-m8mwx\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960554 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhf5\" (UniqueName: \"kubernetes.io/projected/f52c0b21-62cc-4ae5-b309-a79f19fa8974-kube-api-access-wzhf5\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960680 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-k8s-cni-cncf-io\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960717 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-etc-kubernetes\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960746 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960769 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.960783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960796 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-run\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960918 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960946 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-conf\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.960963 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd4aa13b-38c8-4806-978a-0ce17607e28b-data-volume\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961010 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961019 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd4aa13b-38c8-4806-978a-0ce17607e28b-data-volume\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961050 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961116 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961142 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbeceea4-8081-4c42-a085-07b5457ed5c7-host\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961181 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-socket-dir-parent\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961203 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961233 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961252 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961262 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-host\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961274 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961295 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961327 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-socket-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961345 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbbl\" (UniqueName: \"kubernetes.io/projected/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c-kube-api-access-5zbbl\") pod \"kas-connection-checker-685895c7c9-6vb55\" (UID: \"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c\") " pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961430 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-systemd\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961445 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-var-lib-kubelet\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961461 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-cabundle\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961549 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-etc-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961556 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961573 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-netd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961591 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01ed9874-d3cc-4b95-a198-348f68659495-serving-cert\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961620 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-signing-cabundle\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961628 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-trusted-ca\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961658 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-kubelet\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961680 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961696 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-systemd-units\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961724 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961747 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01ed9874-d3cc-4b95-a198-348f68659495-serving-cert\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961832 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-iptables-alerter-script\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961849 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961856 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01ed9874-d3cc-4b95-a198-348f68659495-trusted-ca\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961869 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961868 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwrp\" (UniqueName: \"kubernetes.io/projected/340bdb67-9b1d-40f0-8d58-db1157201e45-kube-api-access-zkwrp\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961932 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-kubernetes\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.961932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961950 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-tmp-dir\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961965 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-tmp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961981 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqr9\" (UniqueName: \"kubernetes.io/projected/e53f8e29-48b9-4952-b819-a2ccd0cc0d60-kube-api-access-fnqr9\") pod \"migrator-66df5d496b-b45zs\" (UID: \"e53f8e29-48b9-4952-b819-a2ccd0cc0d60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.961999 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45sq\" (UniqueName: \"kubernetes.io/projected/fbeceea4-8081-4c42-a085-07b5457ed5c7-kube-api-access-b45sq\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962019 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962058 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f52c0b21-62cc-4ae5-b309-a79f19fa8974-config\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962057 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-tmp-dir\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962068 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8e66c05-354b-4882-97a5-a3c247a548e8-tmp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962071 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-iptables-alerter-script\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962194 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962204 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962248 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-lib-modules\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962279 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwk7k\" (UniqueName: \"kubernetes.io/projected/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-kube-api-access-xwk7k\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962303 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wgp\" (UniqueName: \"kubernetes.io/projected/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-api-access-69wgp\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962308 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f52c0b21-62cc-4ae5-b309-a79f19fa8974-config\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962339 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962348 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0762bfee-d0ac-4a88-bee5-212f3b230a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962357 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-daemon-config\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962374 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962394 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c439b93-fb4b-441d-96d5-54852b26dcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962522 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-tuned\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962546 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-netns\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962573 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962591 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-tuned\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962598 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-modprobe-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962600 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-daemon-config\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962625 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962633 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8e66c05-354b-4882-97a5-a3c247a548e8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962676 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40badafa-2bba-47cc-b94c-c8f7233c7fb6-konnectivity-ca\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962699 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962723 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962814 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.963120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.962824 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40badafa-2bba-47cc-b94c-c8f7233c7fb6-konnectivity-ca\") pod \"konnectivity-agent-mmmgd\" (UID: \"40badafa-2bba-47cc-b94c-c8f7233c7fb6\") " pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:45.974568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.974538 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9vj\" (UniqueName: \"kubernetes.io/projected/4d78fd0a-b2a6-4688-9435-1e5c972575f8-kube-api-access-vk9vj\") pod \"kas-connection-checker-685895c7c9-5p8jz\" (UID: \"4d78fd0a-b2a6-4688-9435-1e5c972575f8\") " pod="kube-system/kas-connection-checker-685895c7c9-5p8jz" Oct 29 22:14:45.974802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.974783 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdjr\" (UniqueName: \"kubernetes.io/projected/547cbb0b-f75f-4a5d-80ee-4d7073cc76f3-kube-api-access-2wdjr\") pod \"network-check-source-6f78d59f99-9xtxc\" (UID: \"547cbb0b-f75f-4a5d-80ee-4d7073cc76f3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xtxc" Oct 29 22:14:45.974869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.974850 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.975287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.975268 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklpp\" (UniqueName: \"kubernetes.io/projected/f8e66c05-354b-4882-97a5-a3c247a548e8-kube-api-access-pklpp\") pod \"insights-operator-7c4986d449-t6pf2\" (UID: \"f8e66c05-354b-4882-97a5-a3c247a548e8\") " pod="openshift-insights/insights-operator-7c4986d449-t6pf2" Oct 29 22:14:45.975453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.975436 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmljm\" (UniqueName: \"kubernetes.io/projected/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-kube-api-access-vmljm\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:45.978061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.978045 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rfk\" (UniqueName: \"kubernetes.io/projected/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-kube-api-access-t7rfk\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:45.978126 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.978112 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79nh\" (UniqueName: \"kubernetes.io/projected/d5414f15-670a-4ca7-a6c2-78a5426d717e-kube-api-access-m79nh\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:45.978937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.978921 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbqh\" (UniqueName: \"kubernetes.io/projected/2e74b42e-7da7-4a53-817b-5a8873b3377d-kube-api-access-8pbqh\") pod \"ingress-canary-97h69\" (UID: \"2e74b42e-7da7-4a53-817b-5a8873b3377d\") " pod="openshift-ingress-canary/ingress-canary-97h69" Oct 29 22:14:45.979095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.979077 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:45.979131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.979076 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww45\" (UniqueName: \"kubernetes.io/projected/c2aaca52-ea07-44d1-8f0f-acb055b79ea3-kube-api-access-bww45\") pod \"service-ca-5fffc97bdf-25css\" (UID: \"c2aaca52-ea07-44d1-8f0f-acb055b79ea3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-25css" Oct 29 22:14:45.979191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.979174 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.979645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.979625 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6vw\" (UniqueName: \"kubernetes.io/projected/49aa972a-477a-419c-a2ba-9e5195039423-kube-api-access-hx6vw\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:45.979750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.979731 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29lf\" (UniqueName: \"kubernetes.io/projected/6813a426-c6a9-4a87-b2b4-8d60472da95d-kube-api-access-n29lf\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:45.980038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.980022 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtmn\" (UniqueName: \"kubernetes.io/projected/0762bfee-d0ac-4a88-bee5-212f3b230a2d-kube-api-access-vqtmn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:45.980114 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.980097 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kwm\" (UniqueName: \"kubernetes.io/projected/c2f1152f-ec5b-4fb7-8b17-04396337eb07-kube-api-access-27kwm\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:45.980114 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.980105 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbbl\" (UniqueName: \"kubernetes.io/projected/3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c-kube-api-access-5zbbl\") pod \"kas-connection-checker-685895c7c9-6vb55\" (UID: \"3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c\") " pod="kube-system/kas-connection-checker-685895c7c9-6vb55" Oct 29 22:14:45.980269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.980254 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfmp\" (UniqueName: \"kubernetes.io/projected/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-kube-api-access-vpfmp\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:45.980728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.980708 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdzn\" (UniqueName: \"kubernetes.io/projected/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-kube-api-access-fwdzn\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:45.981483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.981458 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh6wz\" (UniqueName: \"kubernetes.io/projected/ec88a159-0181-41a7-8d16-fbac28de5674-kube-api-access-bh6wz\") pod \"migrator-66df5d496b-ksr5s\" (UID: \"ec88a159-0181-41a7-8d16-fbac28de5674\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ksr5s" Oct 29 22:14:45.981483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.981478 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbdk\" (UniqueName: \"kubernetes.io/projected/7c439b93-fb4b-441d-96d5-54852b26dcde-kube-api-access-dtbdk\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:45.981608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.981586 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d5z\" (UniqueName: \"kubernetes.io/projected/60db2db0-73c9-4cb7-aa1b-9c670677bc6e-kube-api-access-68d5z\") pod \"kas-connection-checker-685895c7c9-vhdkj\" (UID: \"60db2db0-73c9-4cb7-aa1b-9c670677bc6e\") " pod="kube-system/kas-connection-checker-685895c7c9-vhdkj" Oct 29 22:14:45.981662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.981647 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4278f\" (UniqueName: \"kubernetes.io/projected/a1cc6e96-33dc-4727-b69a-c3050aee425c-kube-api-access-4278f\") pod \"volume-data-source-validator-58c9f5f46d-s4rsk\" (UID: \"a1cc6e96-33dc-4727-b69a-c3050aee425c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s4rsk" Oct 29 22:14:45.981829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.981805 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") pod \"periodic-gathering-n6bxk-w4pqx\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982050 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69wgp\" (UniqueName: \"kubernetes.io/projected/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-api-access-69wgp\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982109 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45sq\" (UniqueName: \"kubernetes.io/projected/fbeceea4-8081-4c42-a085-07b5457ed5c7-kube-api-access-b45sq\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982134 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhf5\" (UniqueName: \"kubernetes.io/projected/f52c0b21-62cc-4ae5-b309-a79f19fa8974-kube-api-access-wzhf5\") pod \"service-ca-operator-544d749c8c-8th7g\" (UID: \"f52c0b21-62cc-4ae5-b309-a79f19fa8974\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982187 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndm85\" (UniqueName: \"kubernetes.io/projected/ce594888-8263-4ccd-b24d-66900da5d257-kube-api-access-ndm85\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4\" (UID: \"ce594888-8263-4ccd-b24d-66900da5d257\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982360 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mwx\" (UniqueName: \"kubernetes.io/projected/01ed9874-d3cc-4b95-a198-348f68659495-kube-api-access-m8mwx\") pod \"console-operator-7cd56b8dbc-wsjxt\" (UID: \"01ed9874-d3cc-4b95-a198-348f68659495\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:45.982613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982581 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwrp\" (UniqueName: \"kubernetes.io/projected/340bdb67-9b1d-40f0-8d58-db1157201e45-kube-api-access-zkwrp\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:45.982852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982643 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct6r\" (UniqueName: \"kubernetes.io/projected/93780727-5cdc-4eaa-91fc-92e7c3889eaa-kube-api-access-xct6r\") pod \"network-check-target-k2gxs\" (UID: \"93780727-5cdc-4eaa-91fc-92e7c3889eaa\") " pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:45.982852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.982794 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqr9\" (UniqueName: \"kubernetes.io/projected/e53f8e29-48b9-4952-b819-a2ccd0cc0d60-kube-api-access-fnqr9\") pod \"migrator-66df5d496b-b45zs\" (UID: \"e53f8e29-48b9-4952-b819-a2ccd0cc0d60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b45zs" Oct 29 22:14:45.983037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:45.983021 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwk7k\" (UniqueName: \"kubernetes.io/projected/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-kube-api-access-xwk7k\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.063645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063598 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:46.063645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063622 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-cnibin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.063645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063645 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-netns\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063664 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340bdb67-9b1d-40f0-8d58-db1157201e45-hosts-file\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063702 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-netns\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063702 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-kubelet\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063715 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-cnibin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063734 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063715 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063729 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-kubelet\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063751 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063752 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340bdb67-9b1d-40f0-8d58-db1157201e45-hosts-file\") pod \"node-resolver-tbx55\" (UID: \"340bdb67-9b1d-40f0-8d58-db1157201e45\") " pod="openshift-dns/node-resolver-tbx55" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063765 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-conf-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.063792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063781 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063802 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-host-slash\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063804 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-conf-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063824 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-var-lib-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063829 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063839 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-system-cni-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063844 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a625d89c-ea34-4bfb-80d6-3bcbf5958f5c-host-slash\") pod \"iptables-alerter-lc8wg\" (UID: \"a625d89c-ea34-4bfb-80d6-3bcbf5958f5c\") " pod="openshift-network-operator/iptables-alerter-lc8wg" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.063855 5073 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063879 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-var-lib-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063863 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-system-cni-dir\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063879 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-systemd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063861 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-systemd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.063931 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.563912509 +0000 UTC m=+1.879306586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063965 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.063994 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-device-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064020 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064048 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064054 5073 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064066 5073 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064073 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-bin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064097 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-multus\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064068 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-device-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064113 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls podName:edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.564096615 +0000 UTC m=+1.879490692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls") pod "dns-default-gxpdb" (UID: "edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3") : secret "dns-default-metrics-tls" not found Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064126 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-bin\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064141 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-cni-multus\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064180 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064196 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064216 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-sys-fs\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064177 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-sys-fs\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064241 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.564229352 +0000 UTC m=+1.879623414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064280 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysconfig\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064303 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-sys\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.064303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064323 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-system-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064362 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysconfig\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064369 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-system-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064372 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-sys\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064387 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-k8s-cni-cncf-io\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064403 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-etc-kubernetes\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064417 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064416 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-k8s-cni-cncf-io\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064431 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-run\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064444 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064446 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-etc-kubernetes\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064464 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-conf\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064482 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064525 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-run\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064535 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064555 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbeceea4-8081-4c42-a085-07b5457ed5c7-host\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064571 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.564560374 +0000 UTC m=+1.879954435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064588 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbeceea4-8081-4c42-a085-07b5457ed5c7-host\") pod \"node-ca-2cvqq\" (UID: \"fbeceea4-8081-4c42-a085-07b5457ed5c7\") " pod="openshift-image-registry/node-ca-2cvqq" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064685 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-socket-dir-parent\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064713 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064728 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064741 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-socket-dir-parent\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064689 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-sysctl-conf\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064743 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-host\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064781 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-socket-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064795 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-systemd\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064808 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-var-lib-kubelet\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064818 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-host\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064827 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-etc-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064778 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.064842 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.564829888 +0000 UTC m=+1.880223977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064865 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-etc-openvswitch\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064883 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-netd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064908 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-systemd\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064912 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-kubelet\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064886 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-socket-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064923 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-var-lib-kubelet\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064948 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-netd\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064957 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-var-lib-kubelet\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064938 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-systemd-units\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.064990 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-systemd-units\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065041 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-kubernetes\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065074 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-lib-modules\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065102 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065125 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-netns\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065176 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-modprobe-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065193 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065198 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-netns\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065205 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065236 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065245 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-lib-modules\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065258 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-multus-certs\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.065263 5073 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065283 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-node-log\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.065307 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.565292751 +0000 UTC m=+1.880686826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065311 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-modprobe-d\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065320 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-node-log\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065337 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-log-socket\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065336 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-host-run-multus-certs\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065366 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-dbus\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065371 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-log-socket\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065390 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-hostroot\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065401 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8709f8-a9fb-4dbe-90ea-2be488f64dc0-etc-kubernetes\") pod \"tuned-5jrfz\" (UID: \"0e8709f8-a9fb-4dbe-90ea-2be488f64dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-5jrfz" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065422 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065438 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-dbus\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065447 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065468 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065473 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-cnibin\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065496 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065521 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-bin\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065548 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-kubelet-config\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065561 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c439b93-fb4b-441d-96d5-54852b26dcde-cnibin\") pod \"multus-additional-cni-plugins-t7vcg\" (UID: \"7c439b93-fb4b-441d-96d5-54852b26dcde\") " pod="openshift-multus/multus-additional-cni-plugins-t7vcg" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065571 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd4aa13b-38c8-4806-978a-0ce17607e28b-crio-socket\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065597 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-etc-selinux\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065614 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-cni-bin\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065623 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-slash\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065633 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c3496-906d-4d23-8e0c-a518b01f9ca3-kubelet-config\") pod \"global-pull-secret-syncer-px4td\" (UID: \"5f6c3496-906d-4d23-8e0c-a518b01f9ca3\") " pod="kube-system/global-pull-secret-syncer-px4td" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065644 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-ovn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065570 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-multus-cni-dir\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065666 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-etc-selinux\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-host-slash\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065704 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0762bfee-d0ac-4a88-bee5-212f3b230a2d-run-ovn\") pod \"ovnkube-node-hx452\" (UID: \"0762bfee-d0ac-4a88-bee5-212f3b230a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065716 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-registration-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065727 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd4aa13b-38c8-4806-978a-0ce17607e28b-crio-socket\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49aa972a-477a-419c-a2ba-9e5195039423-registration-dir\") pod \"aws-ebs-csi-driver-node-8v46d\" (UID: \"49aa972a-477a-419c-a2ba-9e5195039423\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8v46d" Oct 29 22:14:46.065758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065752 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd4aa13b-38c8-4806-978a-0ce17607e28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:46.069046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.065831 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6813a426-c6a9-4a87-b2b4-8d60472da95d-hostroot\") pod \"multus-7qnrm\" (UID: \"6813a426-c6a9-4a87-b2b4-8d60472da95d\") " pod="openshift-multus/multus-7qnrm" Oct 29 22:14:46.069046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.066540 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5414f15-670a-4ca7-a6c2-78a5426d717e-metrics-certs\") pod \"network-metrics-daemon-8j2s2\" (UID: \"d5414f15-670a-4ca7-a6c2-78a5426d717e\") " pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:46.069046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.067053 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"image-registry-84d75cfc68-hctnk\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:46.069046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.067717 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-metrics-certs\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:46.069046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.068047 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"image-registry-6865b464d5-vpmdj\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:46.143919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.143900 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:46.144993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.144965 5073 scope.go:122] "RemoveContainer" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" Oct 29 22:14:46.149227 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.149212 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8j2s2" Oct 29 22:14:46.157211 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.157067 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:46.327279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.327195 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-97h69_2e74b42e-7da7-4a53-817b-5a8873b3377d/serve-healthcheck-canary/0.log" Oct 29 22:14:46.458111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.458086 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8j2s2"] Oct 29 22:14:46.460917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.460896 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84d75cfc68-hctnk"] Oct 29 22:14:46.461729 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:46.461627 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5414f15_670a_4ca7_a6c2_78a5426d717e.slice/crio-fdb85a0ace9eb3a84d930b379ee9bcc857621e34d3dee28bdce15be69c308034 WatchSource:0}: Error finding container fdb85a0ace9eb3a84d930b379ee9bcc857621e34d3dee28bdce15be69c308034: Status 404 returned error can't find the container with id fdb85a0ace9eb3a84d930b379ee9bcc857621e34d3dee28bdce15be69c308034 Oct 29 22:14:46.463723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.463703 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:14:46.464120 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:46.464100 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c2f084_7af8_4fe9_824a_2e50a58eae25.slice/crio-dc49f948972b12cafb84617eac0bd2107bc60cb3fe967573837cd0418a8b052c WatchSource:0}: Error finding container dc49f948972b12cafb84617eac0bd2107bc60cb3fe967573837cd0418a8b052c: Status 404 returned error can't find the container with id dc49f948972b12cafb84617eac0bd2107bc60cb3fe967573837cd0418a8b052c Oct 29 22:14:46.474796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.474777 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6865b464d5-vpmdj"] Oct 29 22:14:46.477757 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:46.477739 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23cc0f7_e165_4cd2_9f5f_06dd5f00fa00.slice/crio-4ca0991605fdbad423d744db1ca5ddf9e9a894635c6b87da3bb5d0bbc8ebc037 WatchSource:0}: Error finding container 4ca0991605fdbad423d744db1ca5ddf9e9a894635c6b87da3bb5d0bbc8ebc037: Status 404 returned error can't find the container with id 4ca0991605fdbad423d744db1ca5ddf9e9a894635c6b87da3bb5d0bbc8ebc037 Oct 29 22:14:46.523336 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.523318 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-b45zs_e53f8e29-48b9-4952-b819-a2ccd0cc0d60/migrator/0.log" Oct 29 22:14:46.571972 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.571942 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:46.572111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.571997 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:46.572111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.572035 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:46.572111 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572068 5073 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:46.572111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.572077 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.572112 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572124 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.572109748 +0000 UTC m=+2.887503819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572178 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.572192 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572232 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.572216939 +0000 UTC m=+2.887611002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572249 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.572240242 +0000 UTC m=+2.887634323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572270 5073 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572274 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572307 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.572296123 +0000 UTC m=+2.887690202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:46.572346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.572325 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:47.572315462 +0000 UTC m=+2.887709572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:46.574438 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.574415 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3-metrics-tls\") pod \"dns-default-gxpdb\" (UID: \"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3\") " pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:46.629438 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.629362 5073 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:46.723762 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.723734 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ksr5s_ec88a159-0181-41a7-8d16-fbac28de5674/migrator/0.log" Oct 29 22:14:46.757845 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.757813 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:46.901129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.901061 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:14:46.901562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.901491 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/1.log" Oct 29 22:14:46.901562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.901518 5073 generic.go:356] "Generic (PLEG): container finished" podID="01ed9874-d3cc-4b95-a198-348f68659495" containerID="cf29e45ae74674fd5c25184ddc1baf673be308796865ee692a6bd383b9c95c01" exitCode=255 Oct 29 22:14:46.901658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.901641 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" event={"ID":"01ed9874-d3cc-4b95-a198-348f68659495","Type":"ContainerDied","Data":"cf29e45ae74674fd5c25184ddc1baf673be308796865ee692a6bd383b9c95c01"} Oct 29 22:14:46.901730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.901714 5073 scope.go:122] "RemoveContainer" containerID="aec6d34d4bfa324ee81f3d344bfc10f0204544ca2c9480065bf6b7e5eef05ba1" Oct 29 22:14:46.901976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.901960 5073 scope.go:122] "RemoveContainer" containerID="cf29e45ae74674fd5c25184ddc1baf673be308796865ee692a6bd383b9c95c01" Oct 29 22:14:46.902218 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:46.902195 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wsjxt_openshift-console-operator(01ed9874-d3cc-4b95-a198-348f68659495)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" Oct 29 22:14:46.903623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.903594 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" event={"ID":"78c2f084-7af8-4fe9-824a-2e50a58eae25","Type":"ContainerStarted","Data":"72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3"} Oct 29 22:14:46.903749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.903648 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" event={"ID":"78c2f084-7af8-4fe9-824a-2e50a58eae25","Type":"ContainerStarted","Data":"dc49f948972b12cafb84617eac0bd2107bc60cb3fe967573837cd0418a8b052c"} Oct 29 22:14:46.905120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.905089 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8j2s2" event={"ID":"d5414f15-670a-4ca7-a6c2-78a5426d717e","Type":"ContainerStarted","Data":"fdb85a0ace9eb3a84d930b379ee9bcc857621e34d3dee28bdce15be69c308034"} Oct 29 22:14:46.907917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.907890 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" event={"ID":"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00","Type":"ContainerStarted","Data":"132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000"} Oct 29 22:14:46.908050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.907933 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" event={"ID":"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00","Type":"ContainerStarted","Data":"4ca0991605fdbad423d744db1ca5ddf9e9a894635c6b87da3bb5d0bbc8ebc037"} Oct 29 22:14:46.925092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:46.925060 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4_ce594888-8263-4ccd-b24d-66900da5d257/kube-storage-version-migrator-operator/0.log" Oct 29 22:14:47.061533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.061505 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gxpdb"] Oct 29 22:14:47.064839 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:47.064811 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc1c3f2_5ad8_4864_bedb_ad117d8fc4a3.slice/crio-c93b685f00fd15554e5794b52d7aa58237b4aca328f5d2fff0be0eb5bd7af558 WatchSource:0}: Error finding container c93b685f00fd15554e5794b52d7aa58237b4aca328f5d2fff0be0eb5bd7af558: Status 404 returned error can't find the container with id c93b685f00fd15554e5794b52d7aa58237b4aca328f5d2fff0be0eb5bd7af558 Oct 29 22:14:47.583760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.583725 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:47.583868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.583792 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:47.583868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.583838 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.583862 5073 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.583917 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls podName:c2f1152f-ec5b-4fb7-8b17-04396337eb07 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.58390298 +0000 UTC m=+4.899297038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zkdq5" (UID: "c2f1152f-ec5b-4fb7-8b17-04396337eb07") : secret "samples-operator-tls" not found Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.583867 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.583935 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.583950 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.583933912 +0000 UTC m=+4.899327972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.583954 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:14:47.583980 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.583976 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.583965387 +0000 UTC m=+4.899359464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:47.584328 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.584007 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert podName:be7200fc-4e5d-416b-8898-2a8bac5648aa nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.583992288 +0000 UTC m=+4.899386349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pcdq8" (UID: "be7200fc-4e5d-416b-8898-2a8bac5648aa") : secret "networking-console-plugin-cert" not found Oct 29 22:14:47.584328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.584060 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:47.584328 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.584190 5073 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:47.584328 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.584232 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls podName:be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:49.584220132 +0000 UTC m=+4.899614197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4n2br" (UID: "be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:14:47.912364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.912327 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxpdb" event={"ID":"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3","Type":"ContainerStarted","Data":"c93b685f00fd15554e5794b52d7aa58237b4aca328f5d2fff0be0eb5bd7af558"} Oct 29 22:14:47.914091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.914071 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:14:47.914530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.914496 5073 scope.go:122] "RemoveContainer" containerID="cf29e45ae74674fd5c25184ddc1baf673be308796865ee692a6bd383b9c95c01" Oct 29 22:14:47.914720 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:47.914700 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wsjxt_openshift-console-operator(01ed9874-d3cc-4b95-a198-348f68659495)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" Oct 29 22:14:47.916399 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.916377 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:47.916480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.916420 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:47.916480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.916377 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8j2s2" event={"ID":"d5414f15-670a-4ca7-a6c2-78a5426d717e","Type":"ContainerStarted","Data":"dccc890a41bde5e3bb9751e9bf66bdfcfb8548ecddb5a985816e1d6baf3463f8"} Oct 29 22:14:47.916644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:47.916516 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8j2s2" event={"ID":"d5414f15-670a-4ca7-a6c2-78a5426d717e","Type":"ContainerStarted","Data":"6983e3a0a26502c327b50f0a6c44c128d80125eea168e9227966f9edbdb92f96"} Oct 29 22:14:48.921140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:48.921061 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxpdb" event={"ID":"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3","Type":"ContainerStarted","Data":"5c810b383eb6dc3bc590f4979b78c284d7e8abe118ac2ed73694f23559c97d01"} Oct 29 22:14:48.921140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:48.921137 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gxpdb" Oct 29 22:14:48.921571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:48.921175 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxpdb" event={"ID":"edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3","Type":"ContainerStarted","Data":"86643f243f429249b1bedc53da35f6e0ab79c37e0bb1e567d3e7a0686615f39e"} Oct 29 22:14:49.603873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.603840 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:49.604061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.603894 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:49.604061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.603924 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:49.604061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.603970 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:49.604061 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:49.603979 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:14:49.604061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.604029 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:49.604061 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:49.604041 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls podName:fd4aa13b-38c8-4806-978a-0ce17607e28b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:53.604026178 +0000 UTC m=+8.919420237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6ccn" (UID: "fd4aa13b-38c8-4806-978a-0ce17607e28b") : secret "insights-runtime-extractor-tls" not found Oct 29 22:14:49.604439 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:49.604089 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:53.604068056 +0000 UTC m=+8.919462145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:49.606786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.606761 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f1152f-ec5b-4fb7-8b17-04396337eb07-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zkdq5\" (UID: \"c2f1152f-ec5b-4fb7-8b17-04396337eb07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:49.606916 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.606820 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4n2br\" (UID: \"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:49.606978 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.606938 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be7200fc-4e5d-416b-8898-2a8bac5648aa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pcdq8\" (UID: \"be7200fc-4e5d-416b-8898-2a8bac5648aa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:49.744969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.744948 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" Oct 29 22:14:49.746122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.746104 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" Oct 29 22:14:49.747544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.747528 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" Oct 29 22:14:49.892253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.892188 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:49.928893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.928865 5073 generic.go:356] "Generic (PLEG): container finished" podID="f52c0b21-62cc-4ae5-b309-a79f19fa8974" containerID="978faa7b7caea8b2056adb61add983d0dd22fa31613814652828b17cae2fbe15" exitCode=0 Oct 29 22:14:49.929220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.928932 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" event={"ID":"f52c0b21-62cc-4ae5-b309-a79f19fa8974","Type":"ContainerDied","Data":"978faa7b7caea8b2056adb61add983d0dd22fa31613814652828b17cae2fbe15"} Oct 29 22:14:49.929357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:49.929342 5073 scope.go:122] "RemoveContainer" containerID="978faa7b7caea8b2056adb61add983d0dd22fa31613814652828b17cae2fbe15" Oct 29 22:14:50.051569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.051541 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br"] Oct 29 22:14:50.054912 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:50.054880 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9c1e3c_0fa7_4509_b4d8_59a57e5fbfd9.slice/crio-b3f54dce776eac0ddd9d3bef119c862ce930a6060ef37e6468ea93b26761bd32 WatchSource:0}: Error finding container b3f54dce776eac0ddd9d3bef119c862ce930a6060ef37e6468ea93b26761bd32: Status 404 returned error can't find the container with id b3f54dce776eac0ddd9d3bef119c862ce930a6060ef37e6468ea93b26761bd32 Oct 29 22:14:50.059131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.059108 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pcdq8"] Oct 29 22:14:50.062632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.062606 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5"] Oct 29 22:14:50.063094 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:50.063068 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7200fc_4e5d_416b_8898_2a8bac5648aa.slice/crio-1c5343f371034bca528ff566bd47e4078d462911f320f2e975d59a5f7f0cde07 WatchSource:0}: Error finding container 1c5343f371034bca528ff566bd47e4078d462911f320f2e975d59a5f7f0cde07: Status 404 returned error can't find the container with id 1c5343f371034bca528ff566bd47e4078d462911f320f2e975d59a5f7f0cde07 Oct 29 22:14:50.510570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.510541 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mmmgd" Oct 29 22:14:50.943261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.943110 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" event={"ID":"c2f1152f-ec5b-4fb7-8b17-04396337eb07","Type":"ContainerStarted","Data":"39fb92743c50bfa8f2cdc691607e5ed41f3b875b6def7dc2117a17e81a5de73c"} Oct 29 22:14:50.944887 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.944851 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" event={"ID":"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9","Type":"ContainerStarted","Data":"b3f54dce776eac0ddd9d3bef119c862ce930a6060ef37e6468ea93b26761bd32"} Oct 29 22:14:50.947891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.947864 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8th7g" event={"ID":"f52c0b21-62cc-4ae5-b309-a79f19fa8974","Type":"ContainerStarted","Data":"b28eabf5f8c470486773c1164b08d3ca135998a8820e372e572f0ece0093b86e"} Oct 29 22:14:50.950540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.950510 5073 generic.go:356] "Generic (PLEG): container finished" podID="ce594888-8263-4ccd-b24d-66900da5d257" containerID="5088e0bf1667b133600a8adb1d80d2501294863e837430277c5c242d7f73668d" exitCode=0 Oct 29 22:14:50.950661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.950607 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" event={"ID":"ce594888-8263-4ccd-b24d-66900da5d257","Type":"ContainerDied","Data":"5088e0bf1667b133600a8adb1d80d2501294863e837430277c5c242d7f73668d"} Oct 29 22:14:50.950935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.950916 5073 scope.go:122] "RemoveContainer" containerID="5088e0bf1667b133600a8adb1d80d2501294863e837430277c5c242d7f73668d" Oct 29 22:14:50.952520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:50.952266 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" event={"ID":"be7200fc-4e5d-416b-8898-2a8bac5648aa","Type":"ContainerStarted","Data":"1c5343f371034bca528ff566bd47e4078d462911f320f2e975d59a5f7f0cde07"} Oct 29 22:14:51.920463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:51.920440 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:14:51.959218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:51.959188 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cvdd4" event={"ID":"ce594888-8263-4ccd-b24d-66900da5d257","Type":"ContainerStarted","Data":"4fa390fba9e4091f9370f3a43ba959470fe6b5c0ff34517fd5f096ae99d03d5c"} Oct 29 22:14:51.961012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:51.960986 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pcdq8" event={"ID":"be7200fc-4e5d-416b-8898-2a8bac5648aa","Type":"ContainerStarted","Data":"e73801c59ba612efa2692e0afc62604b46d8ada7b4d7a961db1a52da542f909f"} Oct 29 22:14:52.078702 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:52.078681 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:14:52.427859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:52.427785 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:14:52.969264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:52.969227 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" event={"ID":"c2f1152f-ec5b-4fb7-8b17-04396337eb07","Type":"ContainerStarted","Data":"98e0b91ff1c8f9b6f00ed7f6c329f943754525385763cafe9fc17867d661e311"} Oct 29 22:14:52.969723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:52.969291 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zkdq5" event={"ID":"c2f1152f-ec5b-4fb7-8b17-04396337eb07","Type":"ContainerStarted","Data":"2a0033ccfe026f0dda69058cbde72adff29afa20451087e7bedfe34de31cd287"} Oct 29 22:14:52.974526 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:52.974487 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4n2br" event={"ID":"be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9","Type":"ContainerStarted","Data":"781be4c4ec743c9b266d60a7c81730e49df2540af88651d53589ee0dfeafb3be"} Oct 29 22:14:53.639847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:53.639799 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:53.639847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:53.639855 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:14:53.640074 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:53.639987 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle podName:7a66a3fc-f006-419b-9442-ef4d8da8bd5e nodeName:}" failed. No retries permitted until 2026-10-29 22:15:01.63997158 +0000 UTC m=+16.955365651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle") pod "router-default-b64c8c7d5-7zxkb" (UID: "7a66a3fc-f006-419b-9442-ef4d8da8bd5e") : configmap references non-existent config key: service-ca.crt Oct 29 22:14:53.642619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:53.642593 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd4aa13b-38c8-4806-978a-0ce17607e28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6ccn\" (UID: \"fd4aa13b-38c8-4806-978a-0ce17607e28b\") " pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:53.659323 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:53.659294 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6ccn" Oct 29 22:14:53.960249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:53.960218 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p6ccn"] Oct 29 22:14:53.963723 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:14:53.963689 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4aa13b_38c8_4806_978a_0ce17607e28b.slice/crio-8152c808ce5744507f21768b1a8bceac86b9e389d017846038a9ae2d953c6a2b WatchSource:0}: Error finding container 8152c808ce5744507f21768b1a8bceac86b9e389d017846038a9ae2d953c6a2b: Status 404 returned error can't find the container with id 8152c808ce5744507f21768b1a8bceac86b9e389d017846038a9ae2d953c6a2b Oct 29 22:14:53.979665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:53.979636 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6ccn" event={"ID":"fd4aa13b-38c8-4806-978a-0ce17607e28b","Type":"ContainerStarted","Data":"8152c808ce5744507f21768b1a8bceac86b9e389d017846038a9ae2d953c6a2b"} Oct 29 22:14:54.985024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:54.984991 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6ccn" event={"ID":"fd4aa13b-38c8-4806-978a-0ce17607e28b","Type":"ContainerStarted","Data":"4cb5b651c62a3a0abbf404ffffe4a3d41284bd27684ea9a989dc93dba75c8ac3"} Oct 29 22:14:54.985360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:54.985047 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6ccn" event={"ID":"fd4aa13b-38c8-4806-978a-0ce17607e28b","Type":"ContainerStarted","Data":"ac4704f3f042d2f835c8997e50a5026526b973d3bded6f347079bfda9a952d25"} Oct 29 22:14:55.367615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:55.367546 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:14:55.367791 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:55.367778 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:56.086271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:56.086236 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:56.086738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:56.086691 5073 scope.go:122] "RemoveContainer" containerID="cf29e45ae74674fd5c25184ddc1baf673be308796865ee692a6bd383b9c95c01" Oct 29 22:14:56.086935 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:14:56.086910 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wsjxt_openshift-console-operator(01ed9874-d3cc-4b95-a198-348f68659495)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" Oct 29 22:14:56.630342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:56.630313 5073 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:56.995693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:56.995612 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6ccn" event={"ID":"fd4aa13b-38c8-4806-978a-0ce17607e28b","Type":"ContainerStarted","Data":"eb6563bd2f09504ca90e2f7d954f24079077ded5c29ecd4477d1750ea7f95f86"} Oct 29 22:14:56.996025 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:56.996005 5073 scope.go:122] "RemoveContainer" containerID="cf29e45ae74674fd5c25184ddc1baf673be308796865ee692a6bd383b9c95c01" Oct 29 22:14:57.277807 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:57.277733 5073 patch_prober.go:29] interesting pod/image-registry-6865b464d5-vpmdj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 29 22:14:57.277807 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:57.277790 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:14:58.002196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:58.002140 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:14:58.002385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:58.002309 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" event={"ID":"01ed9874-d3cc-4b95-a198-348f68659495","Type":"ContainerStarted","Data":"23dc4f0e76171144fbbec2073601e78e9102906445119a6f440671ee4db5cd8a"} Oct 29 22:14:58.002622 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:58.002598 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:58.007508 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:58.007490 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" Oct 29 22:14:58.931302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:14:58.931272 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gxpdb" Oct 29 22:15:01.234377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.234342 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kb25s"] Oct 29 22:15:01.238258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.238231 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v"] Oct 29 22:15:01.238412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.238355 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.241351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.241327 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:01.249410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.249206 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84d75cfc68-hctnk"] Oct 29 22:15:01.257119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.257057 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v"] Oct 29 22:15:01.257287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.257141 5073 patch_prober.go:29] interesting pod/image-registry-84d75cfc68-hctnk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 29 22:15:01.257287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.257215 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" podUID="78c2f084-7af8-4fe9-824a-2e50a58eae25" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:15:01.258065 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.258042 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kb25s"] Oct 29 22:15:01.260714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.260692 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 22:15:01.261586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.261429 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 22:15:01.261586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.261547 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-spjqj\"" Oct 29 22:15:01.268292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.268271 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 29 22:15:01.268418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.268373 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x76j5\"" Oct 29 22:15:01.320202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.320169 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c9b9fb6c4-hlbln"] Oct 29 22:15:01.323418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.323401 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.346309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.346279 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c9b9fb6c4-hlbln"] Oct 29 22:15:01.400690 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.400655 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h2w6v\" (UID: \"2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:01.400895 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.400723 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gjn\" (UniqueName: \"kubernetes.io/projected/0126b13a-114b-474f-8e6d-5142be50e016-kube-api-access-f2gjn\") pod \"downloads-6dd789c8b7-kb25s\" (UID: \"0126b13a-114b-474f-8e6d-5142be50e016\") " pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.400895 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.400823 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0126b13a-114b-474f-8e6d-5142be50e016-tmp\") pod \"downloads-6dd789c8b7-kb25s\" (UID: \"0126b13a-114b-474f-8e6d-5142be50e016\") " pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.501601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501523 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gjn\" (UniqueName: \"kubernetes.io/projected/0126b13a-114b-474f-8e6d-5142be50e016-kube-api-access-f2gjn\") pod \"downloads-6dd789c8b7-kb25s\" (UID: \"0126b13a-114b-474f-8e6d-5142be50e016\") " pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.501601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501562 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0126b13a-114b-474f-8e6d-5142be50e016-tmp\") pod \"downloads-6dd789c8b7-kb25s\" (UID: \"0126b13a-114b-474f-8e6d-5142be50e016\") " pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.501601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501595 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2368862e-24cf-47f7-8627-ccbe846502d0-installation-pull-secrets\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.501867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501636 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-registry-tls\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.501867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501702 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2368862e-24cf-47f7-8627-ccbe846502d0-trusted-ca\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.501867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501761 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-bound-sa-token\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.501867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501819 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwz92\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-kube-api-access-mwz92\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.502006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501907 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2368862e-24cf-47f7-8627-ccbe846502d0-ca-trust-extracted\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.502006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501965 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2368862e-24cf-47f7-8627-ccbe846502d0-image-registry-private-configuration\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.502006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.501998 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h2w6v\" (UID: \"2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:01.502111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.502001 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0126b13a-114b-474f-8e6d-5142be50e016-tmp\") pod \"downloads-6dd789c8b7-kb25s\" (UID: \"0126b13a-114b-474f-8e6d-5142be50e016\") " pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.502111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.502102 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2368862e-24cf-47f7-8627-ccbe846502d0-registry-certificates\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.504746 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.504720 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h2w6v\" (UID: \"2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:01.521959 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.521931 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gjn\" (UniqueName: \"kubernetes.io/projected/0126b13a-114b-474f-8e6d-5142be50e016-kube-api-access-f2gjn\") pod \"downloads-6dd789c8b7-kb25s\" (UID: \"0126b13a-114b-474f-8e6d-5142be50e016\") " pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.562475 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.562453 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:01.571887 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.571845 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:01.603133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603093 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-registry-tls\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603167 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2368862e-24cf-47f7-8627-ccbe846502d0-trusted-ca\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603214 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-bound-sa-token\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603236 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwz92\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-kube-api-access-mwz92\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603260 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2368862e-24cf-47f7-8627-ccbe846502d0-ca-trust-extracted\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603283 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2368862e-24cf-47f7-8627-ccbe846502d0-image-registry-private-configuration\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603341 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2368862e-24cf-47f7-8627-ccbe846502d0-registry-certificates\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.603485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.603436 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2368862e-24cf-47f7-8627-ccbe846502d0-installation-pull-secrets\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.604143 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.604115 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2368862e-24cf-47f7-8627-ccbe846502d0-ca-trust-extracted\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.604441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.604383 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2368862e-24cf-47f7-8627-ccbe846502d0-registry-certificates\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.604629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.604604 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2368862e-24cf-47f7-8627-ccbe846502d0-trusted-ca\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.606249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.606227 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2368862e-24cf-47f7-8627-ccbe846502d0-image-registry-private-configuration\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.606659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.606640 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-registry-tls\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.609427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.609404 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2368862e-24cf-47f7-8627-ccbe846502d0-installation-pull-secrets\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.627012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.626732 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwz92\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-kube-api-access-mwz92\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.631471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.631417 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2368862e-24cf-47f7-8627-ccbe846502d0-bound-sa-token\") pod \"image-registry-5c9b9fb6c4-hlbln\" (UID: \"2368862e-24cf-47f7-8627-ccbe846502d0\") " pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.635656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.635628 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:01.704852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.704823 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:01.705410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.705392 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a66a3fc-f006-419b-9442-ef4d8da8bd5e-service-ca-bundle\") pod \"router-default-b64c8c7d5-7zxkb\" (UID: \"7a66a3fc-f006-419b-9442-ef4d8da8bd5e\") " pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:01.744914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.744889 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:01.884131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.884092 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v"] Oct 29 22:15:01.886883 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.886859 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kb25s"] Oct 29 22:15:01.887131 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:15:01.887103 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fab7257_c4ac_4f52_8fa6_e6f02b12f9cd.slice/crio-6d2a5abe9909fe0b06f166e1b15b75d7694c09f43a1e0aa5d7e7224bf0e90a77 WatchSource:0}: Error finding container 6d2a5abe9909fe0b06f166e1b15b75d7694c09f43a1e0aa5d7e7224bf0e90a77: Status 404 returned error can't find the container with id 6d2a5abe9909fe0b06f166e1b15b75d7694c09f43a1e0aa5d7e7224bf0e90a77 Oct 29 22:15:01.889961 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:15:01.889939 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0126b13a_114b_474f_8e6d_5142be50e016.slice/crio-ba2dad9d385827a0e97010448ca289aa7dac2c32c128ce4de9b1a7f54c877e09 WatchSource:0}: Error finding container ba2dad9d385827a0e97010448ca289aa7dac2c32c128ce4de9b1a7f54c877e09: Status 404 returned error can't find the container with id ba2dad9d385827a0e97010448ca289aa7dac2c32c128ce4de9b1a7f54c877e09 Oct 29 22:15:01.938858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:01.938825 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c9b9fb6c4-hlbln"] Oct 29 22:15:01.942016 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:15:01.941988 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2368862e_24cf_47f7_8627_ccbe846502d0.slice/crio-5f1615ae12662ee77253227830a5fd004485943c057ad52438de931edb8be8fd WatchSource:0}: Error finding container 5f1615ae12662ee77253227830a5fd004485943c057ad52438de931edb8be8fd: Status 404 returned error can't find the container with id 5f1615ae12662ee77253227830a5fd004485943c057ad52438de931edb8be8fd Oct 29 22:15:02.019328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.019261 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kb25s" event={"ID":"0126b13a-114b-474f-8e6d-5142be50e016","Type":"ContainerStarted","Data":"ba2dad9d385827a0e97010448ca289aa7dac2c32c128ce4de9b1a7f54c877e09"} Oct 29 22:15:02.021042 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.021015 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" event={"ID":"2368862e-24cf-47f7-8627-ccbe846502d0","Type":"ContainerStarted","Data":"add4efaff7dd51527183c32873d92c4ecc5a4439940ab54024252d722963caa9"} Oct 29 22:15:02.021144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.021060 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" event={"ID":"2368862e-24cf-47f7-8627-ccbe846502d0","Type":"ContainerStarted","Data":"5f1615ae12662ee77253227830a5fd004485943c057ad52438de931edb8be8fd"} Oct 29 22:15:02.021144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.021127 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:02.022368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.022349 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" event={"ID":"2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd","Type":"ContainerStarted","Data":"6d2a5abe9909fe0b06f166e1b15b75d7694c09f43a1e0aa5d7e7224bf0e90a77"} Oct 29 22:15:02.043848 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.043821 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b64c8c7d5-7zxkb"] Oct 29 22:15:02.046522 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:15:02.046496 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a66a3fc_f006_419b_9442_ef4d8da8bd5e.slice/crio-c289caff886db0530847978874c1a5e0ffd633e54208454ae0c0acba4028c30f WatchSource:0}: Error finding container c289caff886db0530847978874c1a5e0ffd633e54208454ae0c0acba4028c30f: Status 404 returned error can't find the container with id c289caff886db0530847978874c1a5e0ffd633e54208454ae0c0acba4028c30f Oct 29 22:15:02.064375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.064332 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" podStartSLOduration=1.064320501 podStartE2EDuration="1.064320501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.062754028 +0000 UTC m=+17.378148107" watchObservedRunningTime="2026-10-29 22:15:02.064320501 +0000 UTC m=+17.379714580" Oct 29 22:15:02.083326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.083301 5073 patch_prober.go:29] interesting pod/image-registry-6865b464d5-vpmdj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 29 22:15:02.083450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:02.083344 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:15:03.029119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.029079 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" event={"ID":"7a66a3fc-f006-419b-9442-ef4d8da8bd5e","Type":"ContainerStarted","Data":"a613d8eb6e7c56373cd44c1148f999eaedf44158ea92ef99a85a87d9c8c67ea5"} Oct 29 22:15:03.029593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.029137 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" event={"ID":"7a66a3fc-f006-419b-9442-ef4d8da8bd5e","Type":"ContainerStarted","Data":"c289caff886db0530847978874c1a5e0ffd633e54208454ae0c0acba4028c30f"} Oct 29 22:15:03.443554 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.443476 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-966cb74b7-ffhl8"] Oct 29 22:15:03.447078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.447056 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.464002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.463978 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 22:15:03.464128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.464060 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 22:15:03.464128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.464100 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 22:15:03.464231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.464221 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 22:15:03.465562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.465542 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-966cb74b7-ffhl8"] Oct 29 22:15:03.476426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.476364 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 22:15:03.476426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.476397 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rfsln\"" Oct 29 22:15:03.533261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533233 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.533383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533269 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.533383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533289 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.533383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533344 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.533518 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533407 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.533518 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533430 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.533593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.533530 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxr5l\" (UniqueName: \"kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634605 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634576 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634605 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634606 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634741 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxr5l\" (UniqueName: \"kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634785 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634816 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634836 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.634917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.634859 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.635404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.635375 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.635618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.635598 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.635884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.635857 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.635884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.635868 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.637643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.637625 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.638134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.638116 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.657333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.655370 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxr5l\" (UniqueName: \"kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l\") pod \"console-966cb74b7-ffhl8\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:03.745230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.745205 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:03.748018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.747988 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:03.760405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:03.760219 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:04.034137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.034064 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" event={"ID":"2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd","Type":"ContainerStarted","Data":"d82528b802ceda1df7e9e6e541164f21bb7bd865ff4dd198ba2afe7565685741"} Oct 29 22:15:04.034620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.034411 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:04.034620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.034454 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:04.035467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.035430 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b64c8c7d5-7zxkb" Oct 29 22:15:04.039896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.039875 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" Oct 29 22:15:04.062308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.062286 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-966cb74b7-ffhl8"] Oct 29 22:15:04.065875 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:15:04.065853 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dd7cd8_c411_44d7_99e2_0b5c42ea0a65.slice/crio-536829ffe1fa3c3bd0e1802b00b9e6d73cc059ab6eefc2e64c47eb2595e122b9 WatchSource:0}: Error finding container 536829ffe1fa3c3bd0e1802b00b9e6d73cc059ab6eefc2e64c47eb2595e122b9: Status 404 returned error can't find the container with id 536829ffe1fa3c3bd0e1802b00b9e6d73cc059ab6eefc2e64c47eb2595e122b9 Oct 29 22:15:04.073013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.072943 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" podStartSLOduration=1.916862737 podStartE2EDuration="3.072929063s" totalImagesPullingTime="1.156066326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.072449038 +0000 UTC m=+19.387843119" watchObservedRunningTime="2026-10-29 22:15:04.072929063 +0000 UTC m=+19.388323142" Oct 29 22:15:04.329331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:04.329267 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hx452" Oct 29 22:15:05.045566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.045532 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966cb74b7-ffhl8" event={"ID":"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65","Type":"ContainerStarted","Data":"536829ffe1fa3c3bd0e1802b00b9e6d73cc059ab6eefc2e64c47eb2595e122b9"} Oct 29 22:15:05.077938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.077899 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl"] Oct 29 22:15:05.081546 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.081521 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.111771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.111747 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 29 22:15:05.113583 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.113564 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl"] Oct 29 22:15:05.113715 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.113700 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fp22f\"" Oct 29 22:15:05.113834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.113782 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:15:05.113834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.113821 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 29 22:15:05.148819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.147906 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac933a85-a255-47fb-8610-fde57045d702-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.148819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.148003 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac933a85-a255-47fb-8610-fde57045d702-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.148819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.148062 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac933a85-a255-47fb-8610-fde57045d702-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.148819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.148123 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zkf\" (UniqueName: \"kubernetes.io/projected/ac933a85-a255-47fb-8610-fde57045d702-kube-api-access-b4zkf\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.248573 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.248542 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zkf\" (UniqueName: \"kubernetes.io/projected/ac933a85-a255-47fb-8610-fde57045d702-kube-api-access-b4zkf\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.248713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.248602 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac933a85-a255-47fb-8610-fde57045d702-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.248713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.248669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac933a85-a255-47fb-8610-fde57045d702-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.248835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.248720 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac933a85-a255-47fb-8610-fde57045d702-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.249360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.249333 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac933a85-a255-47fb-8610-fde57045d702-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.251577 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.251550 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac933a85-a255-47fb-8610-fde57045d702-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.253760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.253739 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac933a85-a255-47fb-8610-fde57045d702-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.268292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.268264 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zkf\" (UniqueName: \"kubernetes.io/projected/ac933a85-a255-47fb-8610-fde57045d702-kube-api-access-b4zkf\") pod \"prometheus-operator-9cd89b74c-z9xfl\" (UID: \"ac933a85-a255-47fb-8610-fde57045d702\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.399197 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.399112 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" Oct 29 22:15:05.711289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:05.711261 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl"] Oct 29 22:15:06.051286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:06.051198 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" event={"ID":"ac933a85-a255-47fb-8610-fde57045d702","Type":"ContainerStarted","Data":"a1eaf4e23c4797caaf97fb4edd71ea4f14afb7fe2b33ed2639734f6cc01e236f"} Oct 29 22:15:08.064571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:08.064520 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966cb74b7-ffhl8" event={"ID":"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65","Type":"ContainerStarted","Data":"ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0"} Oct 29 22:15:08.068921 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:08.068886 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" event={"ID":"ac933a85-a255-47fb-8610-fde57045d702","Type":"ContainerStarted","Data":"18fd4a7a68acb7fd2a71aeef6b5f3240462b796dbc8ec027ca3669a9dd42e944"} Oct 29 22:15:08.069047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:08.068937 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" event={"ID":"ac933a85-a255-47fb-8610-fde57045d702","Type":"ContainerStarted","Data":"e0a48b22df847c3823a42edf63e5a9ba03a2bad74cbf10450c88bf9a2f76754f"} Oct 29 22:15:08.106259 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:08.106213 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-966cb74b7-ffhl8" podStartSLOduration=1.964483261 podStartE2EDuration="5.106196955s" totalImagesPullingTime="3.141713694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:08.106040643 +0000 UTC m=+23.421434733" watchObservedRunningTime="2026-10-29 22:15:08.106196955 +0000 UTC m=+23.421591033" Oct 29 22:15:08.146958 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:08.146911 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z9xfl" podStartSLOduration=1.603743916 podStartE2EDuration="3.146899397s" totalImagesPullingTime="1.543155481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:08.145642028 +0000 UTC m=+23.461036108" watchObservedRunningTime="2026-10-29 22:15:08.146899397 +0000 UTC m=+23.462293476" Oct 29 22:15:09.259800 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:15:09.259754 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:15:10.714936 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.714906 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s8pn2"] Oct 29 22:15:10.722035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.722013 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.748588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.748528 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:15:10.748588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.748539 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:15:10.751871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.751849 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28glh\"" Oct 29 22:15:10.768044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.768019 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:15:10.801832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.801805 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-sys\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.801996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.801836 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-tls\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.801996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.801859 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.802241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.801995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-wtmp\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.802241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.802061 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a9025bf-fe63-4965-98b5-bace14209a3f-metrics-client-ca\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.802241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.802107 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.802241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.802223 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-root\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.802470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.802255 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-textfile\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.802470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.802344 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0a9025bf-fe63-4965-98b5-bace14209a3f-kube-api-access-lmthp\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903304 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-sys\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903341 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-tls\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903363 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903388 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-wtmp\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903407 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a9025bf-fe63-4965-98b5-bace14209a3f-metrics-client-ca\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903429 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-sys\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903519 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-root\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903848 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903548 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-textfile\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903848 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903549 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-wtmp\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903848 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903571 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0a9025bf-fe63-4965-98b5-bace14209a3f-kube-api-access-lmthp\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.903848 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903603 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a9025bf-fe63-4965-98b5-bace14209a3f-root\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.904125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.903953 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-textfile\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.904125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.904077 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.904125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.904078 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a9025bf-fe63-4965-98b5-bace14209a3f-metrics-client-ca\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.906299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.906277 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-tls\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.910772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.910727 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a9025bf-fe63-4965-98b5-bace14209a3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:10.927385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:10.927364 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0a9025bf-fe63-4965-98b5-bace14209a3f-kube-api-access-lmthp\") pod \"node-exporter-s8pn2\" (UID: \"0a9025bf-fe63-4965-98b5-bace14209a3f\") " pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:11.041370 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:11.041303 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8pn2" Oct 29 22:15:11.093687 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:15:11.093653 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9025bf_fe63_4965_98b5_bace14209a3f.slice/crio-4ee8fadeedc63743dd2b2ddb7aba33a568a5075c4dcb4c7647144b4bc1b0e4dd WatchSource:0}: Error finding container 4ee8fadeedc63743dd2b2ddb7aba33a568a5075c4dcb4c7647144b4bc1b0e4dd: Status 404 returned error can't find the container with id 4ee8fadeedc63743dd2b2ddb7aba33a568a5075c4dcb4c7647144b4bc1b0e4dd Oct 29 22:15:11.255665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:11.255642 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:15:12.084889 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:12.084848 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:15:12.091403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:12.091369 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8pn2" event={"ID":"0a9025bf-fe63-4965-98b5-bace14209a3f","Type":"ContainerStarted","Data":"4ee8fadeedc63743dd2b2ddb7aba33a568a5075c4dcb4c7647144b4bc1b0e4dd"} Oct 29 22:15:13.098369 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:13.098332 5073 generic.go:356] "Generic (PLEG): container finished" podID="0a9025bf-fe63-4965-98b5-bace14209a3f" containerID="26e82d6e5db09e10c0950a21c351e85cff0e39edaccf9e38bbe9a253997adfb8" exitCode=0 Oct 29 22:15:13.098867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:13.098383 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8pn2" event={"ID":"0a9025bf-fe63-4965-98b5-bace14209a3f","Type":"ContainerDied","Data":"26e82d6e5db09e10c0950a21c351e85cff0e39edaccf9e38bbe9a253997adfb8"} Oct 29 22:15:13.761343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:13.761302 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:13.761518 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:13.761397 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:13.767512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:13.767488 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:14.106749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:14.106665 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8pn2" event={"ID":"0a9025bf-fe63-4965-98b5-bace14209a3f","Type":"ContainerStarted","Data":"7ed61615676c119e9167346eb4a70e9ad57c9703cbc6641e95b960b6c988fd23"} Oct 29 22:15:14.106749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:14.106736 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8pn2" event={"ID":"0a9025bf-fe63-4965-98b5-bace14209a3f","Type":"ContainerStarted","Data":"195b5ca081fa3b0392975a69500eff4d14e7588a5684672b452859ad759e285b"} Oct 29 22:15:14.111409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:14.111383 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:14.147773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:14.147724 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s8pn2" podStartSLOduration=2.5020262989999997 podStartE2EDuration="4.147705926s" totalImagesPullingTime="1.542956175s" totalInitContainerRuntime="102.723452ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:14.146363081 +0000 UTC m=+29.461757161" watchObservedRunningTime="2026-10-29 22:15:14.147705926 +0000 UTC m=+29.463100007" Oct 29 22:15:20.138343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:20.138301 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kb25s" event={"ID":"0126b13a-114b-474f-8e6d-5142be50e016","Type":"ContainerStarted","Data":"88dbdb0cd35c77c6ccb02aebf04e5f5ba7919127e9d59f5bafb3c96978b32200"} Oct 29 22:15:20.138860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:20.138524 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:20.153647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:20.153589 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kb25s" Oct 29 22:15:20.205429 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:20.205375 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kb25s" podStartSLOduration=1.9919295529999999 podStartE2EDuration="19.205359659s" totalImagesPullingTime="17.213430106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:20.192090412 +0000 UTC m=+35.507484492" watchObservedRunningTime="2026-10-29 22:15:20.205359659 +0000 UTC m=+35.520753739" Oct 29 22:15:21.923825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:21.923794 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k2gxs" Oct 29 22:15:23.034238 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:23.034202 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c9b9fb6c4-hlbln" Oct 29 22:15:23.125193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:23.125131 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6865b464d5-vpmdj"] Oct 29 22:15:26.279969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.279868 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" podUID="78c2f084-7af8-4fe9-824a-2e50a58eae25" containerName="registry" containerID="cri-o://72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3" gracePeriod=30 Oct 29 22:15:26.600289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.600263 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:15:26.762457 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762426 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762480 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762514 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762543 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762578 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762616 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762903 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762652 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762903 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762693 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") pod \"78c2f084-7af8-4fe9-824a-2e50a58eae25\" (UID: \"78c2f084-7af8-4fe9-824a-2e50a58eae25\") " Oct 29 22:15:26.762903 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762849 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:26.763054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.762912 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:26.763054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.763005 5073 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-certificates\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.763054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.763025 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78c2f084-7af8-4fe9-824a-2e50a58eae25-trusted-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.765867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.765831 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:26.767240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.767213 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:26.767604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.767569 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:26.767745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.767716 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:26.771337 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.771309 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:26.773695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.773666 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66" pod "78c2f084-7af8-4fe9-824a-2e50a58eae25" (UID: "78c2f084-7af8-4fe9-824a-2e50a58eae25"). InnerVolumeSpecName "kube-api-access-pcv66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:26.863773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.863751 5073 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-installation-pull-secrets\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.863870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.863776 5073 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78c2f084-7af8-4fe9-824a-2e50a58eae25-ca-trust-extracted\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.863870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.863785 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcv66\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-kube-api-access-pcv66\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.863870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.863795 5073 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-registry-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.863870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.863805 5073 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78c2f084-7af8-4fe9-824a-2e50a58eae25-image-registry-private-configuration\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:26.863870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:26.863819 5073 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c2f084-7af8-4fe9-824a-2e50a58eae25-bound-sa-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:27.182651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.182519 5073 generic.go:356] "Generic (PLEG): container finished" podID="78c2f084-7af8-4fe9-824a-2e50a58eae25" containerID="72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3" exitCode=0 Oct 29 22:15:27.182651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.182585 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" Oct 29 22:15:27.182863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.182588 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" event={"ID":"78c2f084-7af8-4fe9-824a-2e50a58eae25","Type":"ContainerDied","Data":"72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3"} Oct 29 22:15:27.182863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.182742 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84d75cfc68-hctnk" event={"ID":"78c2f084-7af8-4fe9-824a-2e50a58eae25","Type":"ContainerDied","Data":"dc49f948972b12cafb84617eac0bd2107bc60cb3fe967573837cd0418a8b052c"} Oct 29 22:15:27.182863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.182769 5073 scope.go:122] "RemoveContainer" containerID="72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3" Oct 29 22:15:27.193011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.192989 5073 scope.go:122] "RemoveContainer" containerID="72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3" Oct 29 22:15:27.193336 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:15:27.193306 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3\": container with ID starting with 72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3 not found: ID does not exist" containerID="72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3" Oct 29 22:15:27.193420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.193345 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3"} err="failed to get container status \"72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3\": rpc error: code = NotFound desc = could not find container \"72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3\": container with ID starting with 72be19221a312ab0bbb6ebef6c9d276a8c14ef226c4add6a26dd1fee05527db3 not found: ID does not exist" Oct 29 22:15:27.251990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.251962 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84d75cfc68-hctnk"] Oct 29 22:15:27.265014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:27.264985 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84d75cfc68-hctnk"] Oct 29 22:15:28.836401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:28.836370 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c2f084-7af8-4fe9-824a-2e50a58eae25" path="/var/lib/kubelet/pods/78c2f084-7af8-4fe9-824a-2e50a58eae25/volumes" Oct 29 22:15:34.490180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:34.490138 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-966cb74b7-ffhl8"] Oct 29 22:15:39.253964 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:15:39.253923 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:15:48.155606 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.155556 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerName="registry" containerID="cri-o://132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000" gracePeriod=30 Oct 29 22:15:48.402130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.402102 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:15:48.518355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518328 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518473 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518364 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518518 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518493 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518544 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518569 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518594 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518706 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518663 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.518706 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.518691 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") pod \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\" (UID: \"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00\") " Oct 29 22:15:48.519196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.519169 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:48.519341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.519249 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:48.520837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.520812 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:48.521136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.521117 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:48.521759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.521731 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:48.521867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.521792 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "kube-api-access-lp4tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:48.521931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.521885 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:48.525830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.525809 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted" pod "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" (UID: "a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:48.619834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619812 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-trusted-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619832 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp4tm\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-kube-api-access-lp4tm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619843 5073 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619853 5073 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-installation-pull-secrets\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619862 5073 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-registry-certificates\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619891 5073 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-ca-trust-extracted\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619900 5073 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-image-registry-private-configuration\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:48.619950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:48.619909 5073 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00-bound-sa-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:49.271729 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.271703 5073 generic.go:356] "Generic (PLEG): container finished" podID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerID="132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000" exitCode=0 Oct 29 22:15:49.272103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.271754 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" Oct 29 22:15:49.272103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.271787 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" event={"ID":"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00","Type":"ContainerDied","Data":"132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000"} Oct 29 22:15:49.272103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.271834 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6865b464d5-vpmdj" event={"ID":"a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00","Type":"ContainerDied","Data":"4ca0991605fdbad423d744db1ca5ddf9e9a894635c6b87da3bb5d0bbc8ebc037"} Oct 29 22:15:49.272103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.271856 5073 scope.go:122] "RemoveContainer" containerID="132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000" Oct 29 22:15:49.280170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.280127 5073 scope.go:122] "RemoveContainer" containerID="132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000" Oct 29 22:15:49.280480 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:15:49.280463 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000\": container with ID starting with 132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000 not found: ID does not exist" containerID="132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000" Oct 29 22:15:49.280524 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.280496 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000"} err="failed to get container status \"132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000\": rpc error: code = NotFound desc = could not find container \"132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000\": container with ID starting with 132545016fb7c9b8d41b5ad0ccbf9445e357367ba1b787c0e8e4f97a9497c000 not found: ID does not exist" Oct 29 22:15:49.340907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.338891 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6865b464d5-vpmdj"] Oct 29 22:15:49.350137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:49.350118 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6865b464d5-vpmdj"] Oct 29 22:15:50.836278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:50.836245 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" path="/var/lib/kubelet/pods/a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00/volumes" Oct 29 22:15:59.512238 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.512167 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-966cb74b7-ffhl8" podUID="56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" containerName="console" containerID="cri-o://ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0" gracePeriod=15 Oct 29 22:15:59.813882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.813852 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:15:59.914060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914028 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp\" (UniqueName: \"kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914109 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914140 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914181 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914218 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l\" (UniqueName: \"kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914239 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914290 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert\") pod \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\" (UID: \"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65\") " Oct 29 22:15:59.914522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914352 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:59.914627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914579 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.914627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.914599 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-tmp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:15:59.915239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.915042 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.915239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.915039 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:59.916797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.916774 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "kube-api-access-cxr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:59.916972 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.916954 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:59.917018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:15:59.916974 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert" pod "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" (UID: "56dd7cd8-c411-44d7-99e2-0b5c42ea0a65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.015354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.015318 5073 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-config\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.015354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.015355 5073 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-oauth-config\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.015556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.015371 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.015556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.015387 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxr5l\" (UniqueName: \"kubernetes.io/projected/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-kube-api-access-cxr5l\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.015556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.015401 5073 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-console-serving-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.015556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.015416 5073 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65-oauth-serving-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.321779 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.321741 5073 generic.go:356] "Generic (PLEG): container finished" podID="56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" containerID="ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0" exitCode=0 Oct 29 22:16:00.321949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.321788 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-966cb74b7-ffhl8" Oct 29 22:16:00.321949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.321822 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966cb74b7-ffhl8" event={"ID":"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65","Type":"ContainerDied","Data":"ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0"} Oct 29 22:16:00.321949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.321881 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-966cb74b7-ffhl8" event={"ID":"56dd7cd8-c411-44d7-99e2-0b5c42ea0a65","Type":"ContainerDied","Data":"536829ffe1fa3c3bd0e1802b00b9e6d73cc059ab6eefc2e64c47eb2595e122b9"} Oct 29 22:16:00.321949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.321907 5073 scope.go:122] "RemoveContainer" containerID="ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0" Oct 29 22:16:00.331829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.331811 5073 scope.go:122] "RemoveContainer" containerID="ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0" Oct 29 22:16:00.332114 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:16:00.332092 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0\": container with ID starting with ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0 not found: ID does not exist" containerID="ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0" Oct 29 22:16:00.332183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.332127 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0"} err="failed to get container status \"ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0\": rpc error: code = NotFound desc = could not find container \"ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0\": container with ID starting with ea640133a5c8fef31b399898da818cb88f15642315f08806e8784c1166affbd0 not found: ID does not exist" Oct 29 22:16:00.375186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.375082 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-966cb74b7-ffhl8"] Oct 29 22:16:00.387415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.387391 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-966cb74b7-ffhl8"] Oct 29 22:16:00.837007 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:00.836978 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" path="/var/lib/kubelet/pods/56dd7cd8-c411-44d7-99e2-0b5c42ea0a65/volumes" Oct 29 22:16:04.341074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:04.340983 5073 generic.go:356] "Generic (PLEG): container finished" podID="f8e66c05-354b-4882-97a5-a3c247a548e8" containerID="e7e1a78302b29250ddf2957583e5551ed96d33184a7db9c3a0327666f417f669" exitCode=0 Oct 29 22:16:04.341074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:04.341057 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" event={"ID":"f8e66c05-354b-4882-97a5-a3c247a548e8","Type":"ContainerDied","Data":"e7e1a78302b29250ddf2957583e5551ed96d33184a7db9c3a0327666f417f669"} Oct 29 22:16:04.341604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:04.341387 5073 scope.go:122] "RemoveContainer" containerID="e7e1a78302b29250ddf2957583e5551ed96d33184a7db9c3a0327666f417f669" Oct 29 22:16:04.342758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:04.342732 5073 generic.go:356] "Generic (PLEG): container finished" podID="f7f1df3e-2020-4541-8b82-363ccdad9a05" containerID="1ce7b589dc1e7a1f1248ee320607fea48cf57512ef9b2493d7e9d1d9dae24374" exitCode=0 Oct 29 22:16:04.342872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:04.342771 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" event={"ID":"f7f1df3e-2020-4541-8b82-363ccdad9a05","Type":"ContainerDied","Data":"1ce7b589dc1e7a1f1248ee320607fea48cf57512ef9b2493d7e9d1d9dae24374"} Oct 29 22:16:05.145268 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.145237 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx"] Oct 29 22:16:05.330383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330354 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-27vq7-7ph5s"] Oct 29 22:16:05.330721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330675 5073 cpu_manager.go:414] "containerMap: removing container" podUID="78c2f084-7af8-4fe9-824a-2e50a58eae25" containerName="registry" Oct 29 22:16:05.330721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330686 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c2f084-7af8-4fe9-824a-2e50a58eae25" containerName="registry" Oct 29 22:16:05.330721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330706 5073 cpu_manager.go:414] "containerMap: removing container" podUID="56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" containerName="console" Oct 29 22:16:05.330721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330711 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" containerName="console" Oct 29 22:16:05.330721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330720 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerName="registry" Oct 29 22:16:05.330721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330725 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerName="registry" Oct 29 22:16:05.330948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330773 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23cc0f7-e165-4cd2-9f5f-06dd5f00fa00" containerName="registry" Oct 29 22:16:05.330948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330784 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c2f084-7af8-4fe9-824a-2e50a58eae25" containerName="registry" Oct 29 22:16:05.330948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.330792 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="56dd7cd8-c411-44d7-99e2-0b5c42ea0a65" containerName="console" Oct 29 22:16:05.353710 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.353676 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t6pf2" event={"ID":"f8e66c05-354b-4882-97a5-a3c247a548e8","Type":"ContainerStarted","Data":"a5c3b14d90b29d1c51686802a8eab30b838d19f3f4ae811ac4b04e0f7f1cb747"} Oct 29 22:16:05.354067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.353715 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-27vq7-7ph5s"] Oct 29 22:16:05.354067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.353844 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.462054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.462024 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.462419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.462387 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.462608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.462584 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.462659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.462628 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rkd\" (UniqueName: \"kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.517384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.517360 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:16:05.563208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563172 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") pod \"f7f1df3e-2020-4541-8b82-363ccdad9a05\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " Oct 29 22:16:05.563307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563257 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") pod \"f7f1df3e-2020-4541-8b82-363ccdad9a05\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " Oct 29 22:16:05.563354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563326 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") pod \"f7f1df3e-2020-4541-8b82-363ccdad9a05\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " Oct 29 22:16:05.563400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563362 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") pod \"f7f1df3e-2020-4541-8b82-363ccdad9a05\" (UID: \"f7f1df3e-2020-4541-8b82-363ccdad9a05\") " Oct 29 22:16:05.563500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563481 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.563580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563552 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle" pod "f7f1df3e-2020-4541-8b82-363ccdad9a05" (UID: "f7f1df3e-2020-4541-8b82-363ccdad9a05"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:05.563641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563571 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path" pod "f7f1df3e-2020-4541-8b82-363ccdad9a05" (UID: "f7f1df3e-2020-4541-8b82-363ccdad9a05"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:05.563641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563590 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.563641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563618 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rkd\" (UniqueName: \"kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.563792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563659 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.563792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563703 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle" pod "f7f1df3e-2020-4541-8b82-363ccdad9a05" (UID: "f7f1df3e-2020-4541-8b82-363ccdad9a05"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:05.563792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563715 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-trusted-ca-bundle\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:05.563792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563732 5073 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7f1df3e-2020-4541-8b82-363ccdad9a05-archives-path\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:05.563963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.563889 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.564204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.564187 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.564713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.564691 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.565830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.565807 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd" pod "f7f1df3e-2020-4541-8b82-363ccdad9a05" (UID: "f7f1df3e-2020-4541-8b82-363ccdad9a05"). InnerVolumeSpecName "kube-api-access-p5gpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:05.583288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.583269 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rkd\" (UniqueName: \"kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd\") pod \"periodic-gathering-27vq7-7ph5s\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.664447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.664425 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f1df3e-2020-4541-8b82-363ccdad9a05-service-ca-bundle\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:05.664447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.664445 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gpd\" (UniqueName: \"kubernetes.io/projected/f7f1df3e-2020-4541-8b82-363ccdad9a05-kube-api-access-p5gpd\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:16:05.682430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.682389 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:16:05.995797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:05.995768 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-27vq7-7ph5s"] Oct 29 22:16:05.998768 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:16:05.998744 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ba7300_93bb_4653_b8b5_69b395ed9b06.slice/crio-b1337e0c09de295e3c3daf5dae934cba66dfbf8eebdeacc0090851b9d98c3fa4 WatchSource:0}: Error finding container b1337e0c09de295e3c3daf5dae934cba66dfbf8eebdeacc0090851b9d98c3fa4: Status 404 returned error can't find the container with id b1337e0c09de295e3c3daf5dae934cba66dfbf8eebdeacc0090851b9d98c3fa4 Oct 29 22:16:06.356268 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.356191 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" event={"ID":"f7f1df3e-2020-4541-8b82-363ccdad9a05","Type":"ContainerDied","Data":"4cda4a472c66dfdd7fb1a0fe509e5fd44f020e62cd8fe1e90a3cc9d621be4663"} Oct 29 22:16:06.356268 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.356242 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cda4a472c66dfdd7fb1a0fe509e5fd44f020e62cd8fe1e90a3cc9d621be4663" Oct 29 22:16:06.356268 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.356244 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-n6bxk-w4pqx" Oct 29 22:16:06.358178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.358136 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" event={"ID":"e0ba7300-93bb-4653-b8b5-69b395ed9b06","Type":"ContainerStarted","Data":"9472757ffb6d75c3cc2c1f2bca392c9186e24bad0250ae8f0fe5c44f567da10e"} Oct 29 22:16:06.358326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.358198 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" event={"ID":"e0ba7300-93bb-4653-b8b5-69b395ed9b06","Type":"ContainerStarted","Data":"b1337e0c09de295e3c3daf5dae934cba66dfbf8eebdeacc0090851b9d98c3fa4"} Oct 29 22:16:06.432021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.431952 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" podStartSLOduration=1.431933194 podStartE2EDuration="1.431933194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:06.431542777 +0000 UTC m=+81.746936857" watchObservedRunningTime="2026-10-29 22:16:06.431933194 +0000 UTC m=+81.747327275" Oct 29 22:16:06.467128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.467097 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx"] Oct 29 22:16:06.468796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.468771 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-n6bxk-w4pqx"] Oct 29 22:16:06.836240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:16:06.836209 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f1df3e-2020-4541-8b82-363ccdad9a05" path="/var/lib/kubelet/pods/f7f1df3e-2020-4541-8b82-363ccdad9a05/volumes" Oct 29 22:16:09.254663 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:16:09.254623 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:16:39.253947 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:16:39.253904 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:17:09.253544 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:17:09.253463 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:17:29.723670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:29.723634 5073 generic.go:356] "Generic (PLEG): container finished" podID="e0ba7300-93bb-4653-b8b5-69b395ed9b06" containerID="9472757ffb6d75c3cc2c1f2bca392c9186e24bad0250ae8f0fe5c44f567da10e" exitCode=0 Oct 29 22:17:29.724143 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:29.723697 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" event={"ID":"e0ba7300-93bb-4653-b8b5-69b395ed9b06","Type":"ContainerDied","Data":"9472757ffb6d75c3cc2c1f2bca392c9186e24bad0250ae8f0fe5c44f567da10e"} Oct 29 22:17:30.868311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.868285 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:17:30.915333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915308 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path\") pod \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " Oct 29 22:17:30.915456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915340 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle\") pod \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " Oct 29 22:17:30.915456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915364 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd\" (UniqueName: \"kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd\") pod \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " Oct 29 22:17:30.915456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915412 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle\") pod \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\" (UID: \"e0ba7300-93bb-4653-b8b5-69b395ed9b06\") " Oct 29 22:17:30.915663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915640 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path" pod "e0ba7300-93bb-4653-b8b5-69b395ed9b06" (UID: "e0ba7300-93bb-4653-b8b5-69b395ed9b06"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:30.915803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915777 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle" pod "e0ba7300-93bb-4653-b8b5-69b395ed9b06" (UID: "e0ba7300-93bb-4653-b8b5-69b395ed9b06"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:30.915858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.915843 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle" pod "e0ba7300-93bb-4653-b8b5-69b395ed9b06" (UID: "e0ba7300-93bb-4653-b8b5-69b395ed9b06"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:30.917615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:30.917588 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd" pod "e0ba7300-93bb-4653-b8b5-69b395ed9b06" (UID: "e0ba7300-93bb-4653-b8b5-69b395ed9b06"). InnerVolumeSpecName "kube-api-access-w9rkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:31.016741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.016691 5073 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0ba7300-93bb-4653-b8b5-69b395ed9b06-archives-path\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:17:31.016741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.016711 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-service-ca-bundle\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:17:31.016741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.016720 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rkd\" (UniqueName: \"kubernetes.io/projected/e0ba7300-93bb-4653-b8b5-69b395ed9b06-kube-api-access-w9rkd\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:17:31.016741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.016730 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ba7300-93bb-4653-b8b5-69b395ed9b06-trusted-ca-bundle\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:17:31.735356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.735328 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" Oct 29 22:17:31.735543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.735346 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-27vq7-7ph5s" event={"ID":"e0ba7300-93bb-4653-b8b5-69b395ed9b06","Type":"ContainerDied","Data":"b1337e0c09de295e3c3daf5dae934cba66dfbf8eebdeacc0090851b9d98c3fa4"} Oct 29 22:17:31.735543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:17:31.735429 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1337e0c09de295e3c3daf5dae934cba66dfbf8eebdeacc0090851b9d98c3fa4" Oct 29 22:17:39.252952 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:17:39.252918 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:18:09.257538 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:18:09.257499 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:18:39.257298 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:18:39.257256 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:19:09.254803 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:19:09.254754 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:19:39.257808 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:19:39.257769 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:19:44.855870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:19:44.855840 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:19:44.856389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:19:44.856373 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:19:44.861288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:19:44.861268 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:19:44.861560 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:19:44.861539 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:20:09.256671 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:20:09.256580 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:20:17.487675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.487641 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf"] Oct 29 22:20:17.488057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.487962 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e0ba7300-93bb-4653-b8b5-69b395ed9b06" containerName="insights-gathering" Oct 29 22:20:17.488057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.487973 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ba7300-93bb-4653-b8b5-69b395ed9b06" containerName="insights-gathering" Oct 29 22:20:17.488057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.487982 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f7f1df3e-2020-4541-8b82-363ccdad9a05" containerName="insights-gathering" Oct 29 22:20:17.488057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.487987 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f1df3e-2020-4541-8b82-363ccdad9a05" containerName="insights-gathering" Oct 29 22:20:17.488057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.488056 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ba7300-93bb-4653-b8b5-69b395ed9b06" containerName="insights-gathering" Oct 29 22:20:17.488251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.488063 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f1df3e-2020-4541-8b82-363ccdad9a05" containerName="insights-gathering" Oct 29 22:20:17.491158 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.491132 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.509766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.509743 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf"] Oct 29 22:20:17.579531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.579503 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:20:17.579643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.579593 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:20:17.579715 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.579693 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:20:17.579777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.579731 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kv9sm\"" Oct 29 22:20:17.579777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.579740 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:20:17.579777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.579765 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:20:17.581051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.581034 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e839cdff-da2e-41f3-9755-f15b5a4dfc95-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.581096 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.581080 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e839cdff-da2e-41f3-9755-f15b5a4dfc95-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.581312 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.581122 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.581312 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.581201 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.581312 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.581222 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8j7\" (UniqueName: \"kubernetes.io/projected/e839cdff-da2e-41f3-9755-f15b5a4dfc95-kube-api-access-cl8j7\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.581312 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.581242 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.682523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.682495 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e839cdff-da2e-41f3-9755-f15b5a4dfc95-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.682648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.682568 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e839cdff-da2e-41f3-9755-f15b5a4dfc95-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.682648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.682606 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.682648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.682626 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.682820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.682646 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8j7\" (UniqueName: \"kubernetes.io/projected/e839cdff-da2e-41f3-9755-f15b5a4dfc95-kube-api-access-cl8j7\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.682820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.682676 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.683029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.683005 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e839cdff-da2e-41f3-9755-f15b5a4dfc95-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.685084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.685058 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e839cdff-da2e-41f3-9755-f15b5a4dfc95-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.685208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.685192 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.685373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.685354 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.685432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.685359 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e839cdff-da2e-41f3-9755-f15b5a4dfc95-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.708117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.708088 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8j7\" (UniqueName: \"kubernetes.io/projected/e839cdff-da2e-41f3-9755-f15b5a4dfc95-kube-api-access-cl8j7\") pod \"opendatahub-operator-controller-manager-b95c7d58c-nrbsf\" (UID: \"e839cdff-da2e-41f3-9755-f15b5a4dfc95\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:17.806180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:17.806105 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:18.190656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:18.190633 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf"] Oct 29 22:20:18.192779 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:20:18.192747 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode839cdff_da2e_41f3_9755_f15b5a4dfc95.slice/crio-757d8894e1866fd0aa91b2105d356e16a54f3065e7bb32c44347856ce8b45d1e WatchSource:0}: Error finding container 757d8894e1866fd0aa91b2105d356e16a54f3065e7bb32c44347856ce8b45d1e: Status 404 returned error can't find the container with id 757d8894e1866fd0aa91b2105d356e16a54f3065e7bb32c44347856ce8b45d1e Oct 29 22:20:18.194386 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:18.194360 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:20:18.459087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:18.459052 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" event={"ID":"e839cdff-da2e-41f3-9755-f15b5a4dfc95","Type":"ContainerStarted","Data":"757d8894e1866fd0aa91b2105d356e16a54f3065e7bb32c44347856ce8b45d1e"} Oct 29 22:20:21.475620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:21.475587 5073 generic.go:356] "Generic (PLEG): container finished" podID="e839cdff-da2e-41f3-9755-f15b5a4dfc95" containerID="df0fce09352a022af76206af7733acfa5d9c31ed3ec5a62bce10cf84832e5ff6" exitCode=0 Oct 29 22:20:21.476034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:21.475674 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" event={"ID":"e839cdff-da2e-41f3-9755-f15b5a4dfc95","Type":"ContainerDied","Data":"df0fce09352a022af76206af7733acfa5d9c31ed3ec5a62bce10cf84832e5ff6"} Oct 29 22:20:22.482320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:22.482280 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" event={"ID":"e839cdff-da2e-41f3-9755-f15b5a4dfc95","Type":"ContainerStarted","Data":"8cfe2d5d8dbee320e544ee8f19924201d2c30a8f592fbf8fc325e384f0a5cb56"} Oct 29 22:20:22.482738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:22.482444 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:22.564136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:22.564088 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" podStartSLOduration=2.165179096 podStartE2EDuration="5.564074248s" totalImagesPullingTime="2.90368774s" totalInitContainerRuntime="495.207412ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:22.54932903 +0000 UTC m=+337.864723110" watchObservedRunningTime="2026-10-29 22:20:22.564074248 +0000 UTC m=+337.879468327" Oct 29 22:20:33.489687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:33.489650 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf" Oct 29 22:20:39.255464 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:20:39.255426 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:20:39.510456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.510388 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd"] Oct 29 22:20:39.519490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.519443 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.535075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.535052 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd"] Oct 29 22:20:39.547528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.547505 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 29 22:20:39.547635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.547509 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:20:39.548942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.548918 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nbp\" (UniqueName: \"kubernetes.io/projected/0186fcea-cfec-4721-857e-50d28a8f44b3-kube-api-access-f5nbp\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.548999 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.548979 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0186fcea-cfec-4721-857e-50d28a8f44b3-metrics-cert\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.549125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.549108 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0186fcea-cfec-4721-857e-50d28a8f44b3-manager-config\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.549211 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.549181 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0186fcea-cfec-4721-857e-50d28a8f44b3-cert\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.551079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.551062 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 29 22:20:39.551209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.551195 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 29 22:20:39.551263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.551229 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 29 22:20:39.551263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.551243 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tjxgg\"" Oct 29 22:20:39.649618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.649581 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0186fcea-cfec-4721-857e-50d28a8f44b3-manager-config\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.649800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.649625 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0186fcea-cfec-4721-857e-50d28a8f44b3-cert\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.649800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.649663 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nbp\" (UniqueName: \"kubernetes.io/projected/0186fcea-cfec-4721-857e-50d28a8f44b3-kube-api-access-f5nbp\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.649800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.649694 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0186fcea-cfec-4721-857e-50d28a8f44b3-metrics-cert\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.650420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.650400 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0186fcea-cfec-4721-857e-50d28a8f44b3-manager-config\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.652381 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.652360 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0186fcea-cfec-4721-857e-50d28a8f44b3-metrics-cert\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.652506 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.652428 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0186fcea-cfec-4721-857e-50d28a8f44b3-cert\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.670463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.670444 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nbp\" (UniqueName: \"kubernetes.io/projected/0186fcea-cfec-4721-857e-50d28a8f44b3-kube-api-access-f5nbp\") pod \"lws-controller-manager-76cd858f56-9lpxd\" (UID: \"0186fcea-cfec-4721-857e-50d28a8f44b3\") " pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:39.831024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:39.830943 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:40.128261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:40.128235 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd"] Oct 29 22:20:40.130850 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:20:40.130824 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0186fcea_cfec_4721_857e_50d28a8f44b3.slice/crio-cb3dd3415b310fa0f2e499189b59287a8d109c54d294864dac521393aa905076 WatchSource:0}: Error finding container cb3dd3415b310fa0f2e499189b59287a8d109c54d294864dac521393aa905076: Status 404 returned error can't find the container with id cb3dd3415b310fa0f2e499189b59287a8d109c54d294864dac521393aa905076 Oct 29 22:20:40.564804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:40.564756 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" event={"ID":"0186fcea-cfec-4721-857e-50d28a8f44b3","Type":"ContainerStarted","Data":"cb3dd3415b310fa0f2e499189b59287a8d109c54d294864dac521393aa905076"} Oct 29 22:20:42.999450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:42.999419 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp"] Oct 29 22:20:43.003688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.003665 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.023800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.023203 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:20:43.036568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.036541 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wh8h8\"" Oct 29 22:20:43.038132 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.038100 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp"] Oct 29 22:20:43.073485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073457 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e30adc0-b639-4735-94ab-c80c7d97b200-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073578 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073578 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073518 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97jl\" (UniqueName: \"kubernetes.io/projected/0e30adc0-b639-4735-94ab-c80c7d97b200-kube-api-access-l97jl\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073639 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073686 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073762 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073793 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073872 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.073917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.073889 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174471 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174515 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174530 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174552 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e30adc0-b639-4735-94ab-c80c7d97b200-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174575 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174597 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97jl\" (UniqueName: \"kubernetes.io/projected/0e30adc0-b639-4735-94ab-c80c7d97b200-kube-api-access-l97jl\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174635 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.174836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.175012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.174982 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.175337 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.175205 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.175464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.175331 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.175464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.175409 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e30adc0-b639-4735-94ab-c80c7d97b200-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.175657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.175636 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.177035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.177009 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.177130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.177113 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.195108 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.195087 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e30adc0-b639-4735-94ab-c80c7d97b200-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.195666 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.195649 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97jl\" (UniqueName: \"kubernetes.io/projected/0e30adc0-b639-4735-94ab-c80c7d97b200-kube-api-access-l97jl\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-pzncp\" (UID: \"0e30adc0-b639-4735-94ab-c80c7d97b200\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.319404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.319352 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:43.580013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.579940 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" event={"ID":"0186fcea-cfec-4721-857e-50d28a8f44b3","Type":"ContainerStarted","Data":"4c139a11651cfd157f596126258cf0842e161d757e4374717d5054e7c3580b24"} Oct 29 22:20:43.580162 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.580077 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:43.622942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.622899 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" podStartSLOduration=2.182913238 podStartE2EDuration="4.62288689s" totalImagesPullingTime="2.439973652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:43.622612963 +0000 UTC m=+358.938007053" watchObservedRunningTime="2026-10-29 22:20:43.62288689 +0000 UTC m=+358.938280969" Oct 29 22:20:43.634078 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:20:43.634051 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e30adc0_b639_4735_94ab_c80c7d97b200.slice/crio-4d55da04dd43ade6af874ebca751ef5b9aa98601bf4ced0de86c65a9578defb2 WatchSource:0}: Error finding container 4d55da04dd43ade6af874ebca751ef5b9aa98601bf4ced0de86c65a9578defb2: Status 404 returned error can't find the container with id 4d55da04dd43ade6af874ebca751ef5b9aa98601bf4ced0de86c65a9578defb2 Oct 29 22:20:43.637375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:43.637354 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp"] Oct 29 22:20:44.585971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:44.585923 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" event={"ID":"0e30adc0-b639-4735-94ab-c80c7d97b200","Type":"ContainerStarted","Data":"4d55da04dd43ade6af874ebca751ef5b9aa98601bf4ced0de86c65a9578defb2"} Oct 29 22:20:44.851091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:44.851021 5073 scope.go:122] "RemoveContainer" containerID="1ce7b589dc1e7a1f1248ee320607fea48cf57512ef9b2493d7e9d1d9dae24374" Oct 29 22:20:46.071932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:46.071898 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:20:46.072250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:46.071962 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:20:46.072250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:46.071990 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:20:46.597367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:46.597335 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" event={"ID":"0e30adc0-b639-4735-94ab-c80c7d97b200","Type":"ContainerStarted","Data":"595311ceefe77625fe27d5f5c967c5148a3a2d95bb1c90b07dec5a2e5545bbb6"} Oct 29 22:20:46.641238 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:46.641191 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" podStartSLOduration=2.206224364 podStartE2EDuration="4.641179006s" totalImagesPullingTime="2.434954642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:46.640115735 +0000 UTC m=+361.955509815" watchObservedRunningTime="2026-10-29 22:20:46.641179006 +0000 UTC m=+361.956573085" Oct 29 22:20:47.322247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:47.322200 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:47.326589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:47.326567 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:47.602125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:47.602038 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:47.602909 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:47.602888 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-pzncp" Oct 29 22:20:53.106433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.106401 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk"] Oct 29 22:20:53.109971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.109952 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.127161 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.127123 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pl9nl\"" Oct 29 22:20:53.157645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.157618 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4bda76-7201-4d14-b501-79f936a6fb1b-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.157749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.157649 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889fp\" (UniqueName: \"kubernetes.io/projected/3e4bda76-7201-4d14-b501-79f936a6fb1b-kube-api-access-889fp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.157749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.157675 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e4bda76-7201-4d14-b501-79f936a6fb1b-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.157859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.157829 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e4bda76-7201-4d14-b501-79f936a6fb1b-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.168830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.168809 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 29 22:20:53.170162 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.170134 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk"] Oct 29 22:20:53.258203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.258179 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e4bda76-7201-4d14-b501-79f936a6fb1b-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.258298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.258218 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4bda76-7201-4d14-b501-79f936a6fb1b-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.258298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.258237 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889fp\" (UniqueName: \"kubernetes.io/projected/3e4bda76-7201-4d14-b501-79f936a6fb1b-kube-api-access-889fp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.258298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.258263 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e4bda76-7201-4d14-b501-79f936a6fb1b-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.258644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.258622 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4bda76-7201-4d14-b501-79f936a6fb1b-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.258644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.258639 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e4bda76-7201-4d14-b501-79f936a6fb1b-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.259049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.259032 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e4bda76-7201-4d14-b501-79f936a6fb1b-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.279027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.279005 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889fp\" (UniqueName: \"kubernetes.io/projected/3e4bda76-7201-4d14-b501-79f936a6fb1b-kube-api-access-889fp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lw9wk\" (UID: \"3e4bda76-7201-4d14-b501-79f936a6fb1b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.422975 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.422926 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:53.736810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:53.736787 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk"] Oct 29 22:20:53.738741 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:20:53.738708 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4bda76_7201_4d14_b501_79f936a6fb1b.slice/crio-b6602b3cb11c4d88e01b61b09f6adcda078c1c256aaac593f0ab065e5af0c9b9 WatchSource:0}: Error finding container b6602b3cb11c4d88e01b61b09f6adcda078c1c256aaac593f0ab065e5af0c9b9: Status 404 returned error can't find the container with id b6602b3cb11c4d88e01b61b09f6adcda078c1c256aaac593f0ab065e5af0c9b9 Oct 29 22:20:54.588714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:54.588685 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76cd858f56-9lpxd" Oct 29 22:20:54.637806 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:54.637764 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" event={"ID":"3e4bda76-7201-4d14-b501-79f936a6fb1b","Type":"ContainerStarted","Data":"b6602b3cb11c4d88e01b61b09f6adcda078c1c256aaac593f0ab065e5af0c9b9"} Oct 29 22:20:56.648568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:56.648518 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" event={"ID":"3e4bda76-7201-4d14-b501-79f936a6fb1b","Type":"ContainerStarted","Data":"c988396251c0576f3601df9b8efc4249a525287ea71ecc9202a63c7694607488"} Oct 29 22:20:56.649024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:56.648617 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:20:56.704837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:20:56.704786 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" podStartSLOduration=1.457985347 podStartE2EDuration="3.704773455s" totalImagesPullingTime="2.246788108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:56.703844635 +0000 UTC m=+372.019238714" watchObservedRunningTime="2026-10-29 22:20:56.704773455 +0000 UTC m=+372.020167534" Oct 29 22:21:09.254199 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:21:09.254139 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:21:17.708644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:17.708600 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk" Oct 29 22:21:39.255137 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:21:39.255099 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:21:40.004819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.004785 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9zlw9"] Oct 29 22:21:40.015077 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.015055 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.044024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.044002 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9zlw9"] Oct 29 22:21:40.055601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.055573 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 29 22:21:40.055601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.055585 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h9s2r\"" Oct 29 22:21:40.066306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.066285 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m"] Oct 29 22:21:40.070074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.070050 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.087851 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.087830 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m"] Oct 29 22:21:40.097655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.097631 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dg9sk"] Oct 29 22:21:40.101006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.100986 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.119326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.119305 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dg9sk"] Oct 29 22:21:40.131426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.131404 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30070c77-9435-4c71-b525-5fbc18e94ed7-cert\") pod \"kserve-controller-manager-776cc66d47-9zlw9\" (UID: \"30070c77-9435-4c71-b525-5fbc18e94ed7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.131537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.131438 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjtq\" (UniqueName: \"kubernetes.io/projected/30070c77-9435-4c71-b525-5fbc18e94ed7-kube-api-access-jrjtq\") pod \"kserve-controller-manager-776cc66d47-9zlw9\" (UID: \"30070c77-9435-4c71-b525-5fbc18e94ed7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.141678 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.141652 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-2fxfz"] Oct 29 22:21:40.141808 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.141787 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 29 22:21:40.141959 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.141944 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bphz5\"" Oct 29 22:21:40.142037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.141953 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-wvljb\"" Oct 29 22:21:40.142037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.141949 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 29 22:21:40.147122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.147104 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.164046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.164023 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-2fxfz"] Oct 29 22:21:40.232533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.232513 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30070c77-9435-4c71-b525-5fbc18e94ed7-cert\") pod \"kserve-controller-manager-776cc66d47-9zlw9\" (UID: \"30070c77-9435-4c71-b525-5fbc18e94ed7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.232648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.232546 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjtq\" (UniqueName: \"kubernetes.io/projected/30070c77-9435-4c71-b525-5fbc18e94ed7-kube-api-access-jrjtq\") pod \"kserve-controller-manager-776cc66d47-9zlw9\" (UID: \"30070c77-9435-4c71-b525-5fbc18e94ed7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.232648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.232579 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mgc6m\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.232648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.232616 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnz2r\" (UniqueName: \"kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r\") pod \"llmisvc-controller-manager-869d5dcc5d-mgc6m\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.232810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.232648 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5nv\" (UniqueName: \"kubernetes.io/projected/a0e90837-ba5f-4c9d-9f0c-839609878ed9-kube-api-access-sv5nv\") pod \"model-serving-api-75bd89fdfc-dg9sk\" (UID: \"a0e90837-ba5f-4c9d-9f0c-839609878ed9\") " pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.232810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.232676 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e90837-ba5f-4c9d-9f0c-839609878ed9-tls-certs\") pod \"model-serving-api-75bd89fdfc-dg9sk\" (UID: \"a0e90837-ba5f-4c9d-9f0c-839609878ed9\") " pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.235077 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.235047 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30070c77-9435-4c71-b525-5fbc18e94ed7-cert\") pod \"kserve-controller-manager-776cc66d47-9zlw9\" (UID: \"30070c77-9435-4c71-b525-5fbc18e94ed7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.241821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.241797 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 29 22:21:40.241990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.241973 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pq2zk\"" Oct 29 22:21:40.242069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.242056 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 29 22:21:40.252770 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.252747 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjtq\" (UniqueName: \"kubernetes.io/projected/30070c77-9435-4c71-b525-5fbc18e94ed7-kube-api-access-jrjtq\") pod \"kserve-controller-manager-776cc66d47-9zlw9\" (UID: \"30070c77-9435-4c71-b525-5fbc18e94ed7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.327654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.327602 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:40.334014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.333996 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnz2r\" (UniqueName: \"kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r\") pod \"llmisvc-controller-manager-869d5dcc5d-mgc6m\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.334084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.334032 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5nv\" (UniqueName: \"kubernetes.io/projected/a0e90837-ba5f-4c9d-9f0c-839609878ed9-kube-api-access-sv5nv\") pod \"model-serving-api-75bd89fdfc-dg9sk\" (UID: \"a0e90837-ba5f-4c9d-9f0c-839609878ed9\") " pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.334084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.334073 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slcdr\" (UniqueName: \"kubernetes.io/projected/5ce9a9c1-c283-4138-b25b-f4bff5aee593-kube-api-access-slcdr\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.334181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.334099 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e90837-ba5f-4c9d-9f0c-839609878ed9-tls-certs\") pod \"model-serving-api-75bd89fdfc-dg9sk\" (UID: \"a0e90837-ba5f-4c9d-9f0c-839609878ed9\") " pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.334181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.334125 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ce9a9c1-c283-4138-b25b-f4bff5aee593-cert\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.334259 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.334223 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce9a9c1-c283-4138-b25b-f4bff5aee593-metrics-certs\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.334296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.334277 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mgc6m\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.336437 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.336416 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e90837-ba5f-4c9d-9f0c-839609878ed9-tls-certs\") pod \"model-serving-api-75bd89fdfc-dg9sk\" (UID: \"a0e90837-ba5f-4c9d-9f0c-839609878ed9\") " pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.337529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.337504 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mgc6m\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.354684 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.354659 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5nv\" (UniqueName: \"kubernetes.io/projected/a0e90837-ba5f-4c9d-9f0c-839609878ed9-kube-api-access-sv5nv\") pod \"model-serving-api-75bd89fdfc-dg9sk\" (UID: \"a0e90837-ba5f-4c9d-9f0c-839609878ed9\") " pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.356603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.356570 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnz2r\" (UniqueName: \"kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r\") pod \"llmisvc-controller-manager-869d5dcc5d-mgc6m\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.382622 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.382495 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:40.417089 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.417058 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:40.435693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.435658 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce9a9c1-c283-4138-b25b-f4bff5aee593-metrics-certs\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.435816 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.435797 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slcdr\" (UniqueName: \"kubernetes.io/projected/5ce9a9c1-c283-4138-b25b-f4bff5aee593-kube-api-access-slcdr\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.435879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.435848 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ce9a9c1-c283-4138-b25b-f4bff5aee593-cert\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.443045 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.442784 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ce9a9c1-c283-4138-b25b-f4bff5aee593-cert\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.444879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.444827 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce9a9c1-c283-4138-b25b-f4bff5aee593-metrics-certs\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.458119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.458075 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slcdr\" (UniqueName: \"kubernetes.io/projected/5ce9a9c1-c283-4138-b25b-f4bff5aee593-kube-api-access-slcdr\") pod \"odh-model-controller-f87fcf56c-2fxfz\" (UID: \"5ce9a9c1-c283-4138-b25b-f4bff5aee593\") " pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.460957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.460928 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:40.627254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.627222 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9zlw9"] Oct 29 22:21:40.628829 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:21:40.628808 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30070c77_9435_4c71_b525_5fbc18e94ed7.slice/crio-f87caeae1baef4955b576232a3ca26fa4ccb17fb9faeffb3c6042e42a33dc3f5 WatchSource:0}: Error finding container f87caeae1baef4955b576232a3ca26fa4ccb17fb9faeffb3c6042e42a33dc3f5: Status 404 returned error can't find the container with id f87caeae1baef4955b576232a3ca26fa4ccb17fb9faeffb3c6042e42a33dc3f5 Oct 29 22:21:40.687540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.687521 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m"] Oct 29 22:21:40.689076 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:21:40.689050 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf4ce2eb_e9fd_4a21_93cf_0071ff28b9bb.slice/crio-d2c8358b88e5a28b48349c9d18736383c61d0e09cef18401bfa197412c3c5296 WatchSource:0}: Error finding container d2c8358b88e5a28b48349c9d18736383c61d0e09cef18401bfa197412c3c5296: Status 404 returned error can't find the container with id d2c8358b88e5a28b48349c9d18736383c61d0e09cef18401bfa197412c3c5296 Oct 29 22:21:40.723483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.723452 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dg9sk"] Oct 29 22:21:40.724545 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:21:40.724525 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e90837_ba5f_4c9d_9f0c_839609878ed9.slice/crio-7139c3f89b4b23f6be36c416cf2d94b3e14eb8ebf2764cf2fd9e9c3b6c70fff5 WatchSource:0}: Error finding container 7139c3f89b4b23f6be36c416cf2d94b3e14eb8ebf2764cf2fd9e9c3b6c70fff5: Status 404 returned error can't find the container with id 7139c3f89b4b23f6be36c416cf2d94b3e14eb8ebf2764cf2fd9e9c3b6c70fff5 Oct 29 22:21:40.772833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.772815 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-2fxfz"] Oct 29 22:21:40.774705 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:21:40.774683 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce9a9c1_c283_4138_b25b_f4bff5aee593.slice/crio-ef8f1cea08a61127a1f20c757daade5e07f74509d87cba072245029d94b8ecb3 WatchSource:0}: Error finding container ef8f1cea08a61127a1f20c757daade5e07f74509d87cba072245029d94b8ecb3: Status 404 returned error can't find the container with id ef8f1cea08a61127a1f20c757daade5e07f74509d87cba072245029d94b8ecb3 Oct 29 22:21:40.844433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.844407 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" event={"ID":"a0e90837-ba5f-4c9d-9f0c-839609878ed9","Type":"ContainerStarted","Data":"7139c3f89b4b23f6be36c416cf2d94b3e14eb8ebf2764cf2fd9e9c3b6c70fff5"} Oct 29 22:21:40.845693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.845662 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" event={"ID":"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb","Type":"ContainerStarted","Data":"d2c8358b88e5a28b48349c9d18736383c61d0e09cef18401bfa197412c3c5296"} Oct 29 22:21:40.846792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.846771 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" event={"ID":"30070c77-9435-4c71-b525-5fbc18e94ed7","Type":"ContainerStarted","Data":"f87caeae1baef4955b576232a3ca26fa4ccb17fb9faeffb3c6042e42a33dc3f5"} Oct 29 22:21:40.847851 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:40.847830 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" event={"ID":"5ce9a9c1-c283-4138-b25b-f4bff5aee593","Type":"ContainerStarted","Data":"ef8f1cea08a61127a1f20c757daade5e07f74509d87cba072245029d94b8ecb3"} Oct 29 22:21:43.869392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.869323 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" event={"ID":"a0e90837-ba5f-4c9d-9f0c-839609878ed9","Type":"ContainerStarted","Data":"cb15e01c5453808e227ce5484b621b795e098a00f6b50173bc64502feb24cede"} Oct 29 22:21:43.869818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.869501 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:43.872905 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.872877 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" event={"ID":"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb","Type":"ContainerStarted","Data":"a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7"} Oct 29 22:21:43.873028 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.873020 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:21:43.874685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.874656 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" event={"ID":"30070c77-9435-4c71-b525-5fbc18e94ed7","Type":"ContainerStarted","Data":"ae2225cf2afd01b6656fce801279aa622a78dc7103f2aa7d885a0b6a9bdc10af"} Oct 29 22:21:43.874804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.874763 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:21:43.913815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.913763 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" podStartSLOduration=1.421242981 podStartE2EDuration="3.913745255s" totalImagesPullingTime="2.492502274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:43.910735566 +0000 UTC m=+419.226129646" watchObservedRunningTime="2026-10-29 22:21:43.913745255 +0000 UTC m=+419.229139338" Oct 29 22:21:43.951333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.951278 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" podStartSLOduration=1.28139359 podStartE2EDuration="3.951265257s" totalImagesPullingTime="2.669871667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:43.949934493 +0000 UTC m=+419.265328573" watchObservedRunningTime="2026-10-29 22:21:43.951265257 +0000 UTC m=+419.266659336" Oct 29 22:21:43.991365 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:43.991314 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" podStartSLOduration=1.851378634 podStartE2EDuration="4.991300603s" totalImagesPullingTime="3.139921969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:43.990656685 +0000 UTC m=+419.306050767" watchObservedRunningTime="2026-10-29 22:21:43.991300603 +0000 UTC m=+419.306694682" Oct 29 22:21:44.879837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:44.879806 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" event={"ID":"5ce9a9c1-c283-4138-b25b-f4bff5aee593","Type":"ContainerStarted","Data":"65a27057213ed5e3cb96d1ad880b549f617060e6e0bc2db54f49cda8f5c569c5"} Oct 29 22:21:44.880254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:44.879866 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:21:44.920913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:44.920875 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" podStartSLOduration=1.403978524 podStartE2EDuration="4.920864227s" totalImagesPullingTime="3.516885703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:44.920267091 +0000 UTC m=+420.235661195" watchObservedRunningTime="2026-10-29 22:21:44.920864227 +0000 UTC m=+420.236258306" Oct 29 22:21:54.883926 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:54.883892 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-dg9sk" Oct 29 22:21:55.886731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:21:55.886703 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-2fxfz" Oct 29 22:22:09.254119 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:22:09.254081 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:22:14.882609 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:22:14.882576 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:22:14.885802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:22:14.885781 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9zlw9" Oct 29 22:22:39.256988 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:22:39.256947 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:23:09.254716 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:23:09.254639 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:23:39.255505 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:23:39.255462 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:23:41.621065 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.621034 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr"] Oct 29 22:23:41.626812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.626789 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.645279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.645254 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lf5dq\"" Oct 29 22:23:41.660557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.660529 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr"] Oct 29 22:23:41.679981 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.679956 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680005 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680077 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680211 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680252 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nml6\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680307 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680385 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680444 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680414 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.680497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.680481 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.780865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.780841 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.780873 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nml6\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.780893 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.780922 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.780947 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.780972 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781020 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781054 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781076 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781279 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781288 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781354 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.781614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.781598 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.782202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.782178 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.783556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.783530 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.783670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.783652 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.801105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.801084 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.801257 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.801241 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nml6\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:41.941648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:41.941593 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:42.256543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.256521 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr"] Oct 29 22:23:42.257836 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:23:42.257809 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eb11cf_7c12_4856_8605_88463865fcd4.slice/crio-e850d4f2ff87c599f837395a3b9c198e91225b5bd7bd80eafea792a0a916bda9 WatchSource:0}: Error finding container e850d4f2ff87c599f837395a3b9c198e91225b5bd7bd80eafea792a0a916bda9: Status 404 returned error can't find the container with id e850d4f2ff87c599f837395a3b9c198e91225b5bd7bd80eafea792a0a916bda9 Oct 29 22:23:42.259854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.259814 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:23:42.259964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.259897 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:23:42.259964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.259944 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:23:42.411878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.411837 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" event={"ID":"56eb11cf-7c12-4856-8605-88463865fcd4","Type":"ContainerStarted","Data":"a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805"} Oct 29 22:23:42.412051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.411896 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" event={"ID":"56eb11cf-7c12-4856-8605-88463865fcd4","Type":"ContainerStarted","Data":"e850d4f2ff87c599f837395a3b9c198e91225b5bd7bd80eafea792a0a916bda9"} Oct 29 22:23:42.455828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.455771 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podStartSLOduration=1.455757545 podStartE2EDuration="1.455757545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:42.453919416 +0000 UTC m=+537.769313496" watchObservedRunningTime="2026-10-29 22:23:42.455757545 +0000 UTC m=+537.771151626" Oct 29 22:23:42.942714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:42.942683 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:43.947604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:43.947569 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:44.421723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:44.421683 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:23:44.422801 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:23:44.422780 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:24:00.391086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.391048 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:24:00.394746 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.394727 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.411984 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.411957 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:24:00.424003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.423977 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 22:24:00.424134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.424032 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:24:00.424134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.424046 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:24:00.425099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.425050 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kzfft\"" Oct 29 22:24:00.468486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.468465 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:24:00.524972 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.524943 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68657adb-f9ad-417d-a37d-2252e0fecb41-config-file\") pod \"limitador-limitador-7c9b87d574-pw8wr\" (UID: \"68657adb-f9ad-417d-a37d-2252e0fecb41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.525064 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.524983 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrcn\" (UniqueName: \"kubernetes.io/projected/68657adb-f9ad-417d-a37d-2252e0fecb41-kube-api-access-zvrcn\") pod \"limitador-limitador-7c9b87d574-pw8wr\" (UID: \"68657adb-f9ad-417d-a37d-2252e0fecb41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.625337 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.625315 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68657adb-f9ad-417d-a37d-2252e0fecb41-config-file\") pod \"limitador-limitador-7c9b87d574-pw8wr\" (UID: \"68657adb-f9ad-417d-a37d-2252e0fecb41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.625436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.625352 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrcn\" (UniqueName: \"kubernetes.io/projected/68657adb-f9ad-417d-a37d-2252e0fecb41-kube-api-access-zvrcn\") pod \"limitador-limitador-7c9b87d574-pw8wr\" (UID: \"68657adb-f9ad-417d-a37d-2252e0fecb41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.625941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.625921 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68657adb-f9ad-417d-a37d-2252e0fecb41-config-file\") pod \"limitador-limitador-7c9b87d574-pw8wr\" (UID: \"68657adb-f9ad-417d-a37d-2252e0fecb41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.646580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.646526 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrcn\" (UniqueName: \"kubernetes.io/projected/68657adb-f9ad-417d-a37d-2252e0fecb41-kube-api-access-zvrcn\") pod \"limitador-limitador-7c9b87d574-pw8wr\" (UID: \"68657adb-f9ad-417d-a37d-2252e0fecb41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:00.708464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:00.708441 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:01.006123 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:01.006100 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:24:01.008060 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:24:01.008036 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68657adb_f9ad_417d_a37d_2252e0fecb41.slice/crio-d446def74a186482051384c4b53bc50244e04c48d4604e9a42cc879efeacd345 WatchSource:0}: Error finding container d446def74a186482051384c4b53bc50244e04c48d4604e9a42cc879efeacd345: Status 404 returned error can't find the container with id d446def74a186482051384c4b53bc50244e04c48d4604e9a42cc879efeacd345 Oct 29 22:24:01.500530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:01.500424 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" event={"ID":"68657adb-f9ad-417d-a37d-2252e0fecb41","Type":"ContainerStarted","Data":"d446def74a186482051384c4b53bc50244e04c48d4604e9a42cc879efeacd345"} Oct 29 22:24:03.511234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:03.511199 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" event={"ID":"68657adb-f9ad-417d-a37d-2252e0fecb41","Type":"ContainerStarted","Data":"24dd13d19066e02d96e7a5dbc2fe56d6c022ec72cf5de2c480ee29b81b0ade24"} Oct 29 22:24:03.511579 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:03.511332 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:03.554982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:03.554941 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" podStartSLOduration=1.502065269 podStartE2EDuration="3.554931285s" totalImagesPullingTime="2.052866016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:03.553845754 +0000 UTC m=+558.869239834" watchObservedRunningTime="2026-10-29 22:24:03.554931285 +0000 UTC m=+558.870325364" Oct 29 22:24:09.255068 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:24:09.254964 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:24:14.517387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:14.517352 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pw8wr" Oct 29 22:24:39.255589 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:24:39.255552 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:24:44.887710 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:44.887685 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:24:44.888185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:44.888060 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:24:44.892899 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:44.892877 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:24:44.893397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:24:44.893371 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:25:09.256181 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:25:09.256124 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:25:20.624250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:20.624219 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:25:29.045165 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.045118 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt"] Oct 29 22:25:29.048773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.048755 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.065954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.065926 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551636ca-bee8-45ca-96b3-0e57666a5e6c-cert\") pod \"llmisvc-controller-manager-6466c554d8-ldwdt\" (UID: \"551636ca-bee8-45ca-96b3-0e57666a5e6c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.066069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.065995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsd2\" (UniqueName: \"kubernetes.io/projected/551636ca-bee8-45ca-96b3-0e57666a5e6c-kube-api-access-xxsd2\") pod \"llmisvc-controller-manager-6466c554d8-ldwdt\" (UID: \"551636ca-bee8-45ca-96b3-0e57666a5e6c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.066769 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.066749 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt"] Oct 29 22:25:29.166893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.166862 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsd2\" (UniqueName: \"kubernetes.io/projected/551636ca-bee8-45ca-96b3-0e57666a5e6c-kube-api-access-xxsd2\") pod \"llmisvc-controller-manager-6466c554d8-ldwdt\" (UID: \"551636ca-bee8-45ca-96b3-0e57666a5e6c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.167049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.166958 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551636ca-bee8-45ca-96b3-0e57666a5e6c-cert\") pod \"llmisvc-controller-manager-6466c554d8-ldwdt\" (UID: \"551636ca-bee8-45ca-96b3-0e57666a5e6c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.169544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.169519 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551636ca-bee8-45ca-96b3-0e57666a5e6c-cert\") pod \"llmisvc-controller-manager-6466c554d8-ldwdt\" (UID: \"551636ca-bee8-45ca-96b3-0e57666a5e6c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.187470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.187447 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsd2\" (UniqueName: \"kubernetes.io/projected/551636ca-bee8-45ca-96b3-0e57666a5e6c-kube-api-access-xxsd2\") pod \"llmisvc-controller-manager-6466c554d8-ldwdt\" (UID: \"551636ca-bee8-45ca-96b3-0e57666a5e6c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.362346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.362268 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:29.661071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.661050 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt"] Oct 29 22:25:29.663481 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:25:29.663457 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551636ca_bee8_45ca_96b3_0e57666a5e6c.slice/crio-2c1dd2fef5afb425aae0fd55941f9d08f1de18c32b8243f805a1d1dbd5e4f527 WatchSource:0}: Error finding container 2c1dd2fef5afb425aae0fd55941f9d08f1de18c32b8243f805a1d1dbd5e4f527: Status 404 returned error can't find the container with id 2c1dd2fef5afb425aae0fd55941f9d08f1de18c32b8243f805a1d1dbd5e4f527 Oct 29 22:25:29.665533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.665511 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:25:29.907334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.906929 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:54266->10.132.0.42:15021: read: connection reset by peer" start-of-body= Oct 29 22:25:29.907334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.906980 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:54266->10.132.0.42:15021: read: connection reset by peer" Oct 29 22:25:29.944066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:29.944035 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" event={"ID":"551636ca-bee8-45ca-96b3-0e57666a5e6c","Type":"ContainerStarted","Data":"2c1dd2fef5afb425aae0fd55941f9d08f1de18c32b8243f805a1d1dbd5e4f527"} Oct 29 22:25:30.948967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.948939 5073 generic.go:356] "Generic (PLEG): container finished" podID="56eb11cf-7c12-4856-8605-88463865fcd4" containerID="a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805" exitCode=0 Oct 29 22:25:30.949421 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.949016 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" event={"ID":"56eb11cf-7c12-4856-8605-88463865fcd4","Type":"ContainerDied","Data":"a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805"} Oct 29 22:25:30.949421 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.949332 5073 scope.go:122] "RemoveContainer" containerID="a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805" Oct 29 22:25:30.950407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.950365 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:25:30.950520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.950453 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:25:30.950520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.950489 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:25:30.950739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.950716 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" event={"ID":"551636ca-bee8-45ca-96b3-0e57666a5e6c","Type":"ContainerStarted","Data":"44d75eba4837759ff56252709d7a9ab587c1699f4e0a7628932bbe754c5d42f3"} Oct 29 22:25:30.950912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:30.950849 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:25:31.049047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:31.048990 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" podStartSLOduration=1.683738345 podStartE2EDuration="2.048976101s" totalImagesPullingTime="365.237756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:31.048293846 +0000 UTC m=+646.363687944" watchObservedRunningTime="2026-10-29 22:25:31.048976101 +0000 UTC m=+646.364370180" Oct 29 22:25:31.958626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:31.958589 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" event={"ID":"56eb11cf-7c12-4856-8605-88463865fcd4","Type":"ContainerStarted","Data":"6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3"} Oct 29 22:25:32.942328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:32.942289 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:25:32.942511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:32.942472 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:25:32.942558 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:32.942524 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 29 22:25:33.942608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:33.942572 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:25:33.943006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:33.942623 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 29 22:25:34.942658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:34.942627 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:25:34.943035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:34.942683 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 29 22:25:35.942697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:35.942660 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:25:35.943087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:35.942741 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 29 22:25:36.945720 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:36.945637 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:25:36.946223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:36.946036 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:25:36.946655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:25:36.946638 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:25:39.256871 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:25:39.256829 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:26:01.960979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:01.960894 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt" Oct 29 22:26:02.026783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.026748 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m"] Oct 29 22:26:02.097465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.097431 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" podUID="df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" containerName="manager" containerID="cri-o://a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7" gracePeriod=30 Oct 29 22:26:02.361598 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.361572 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:26:02.459092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.459056 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r\" (UniqueName: \"kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r\") pod \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " Oct 29 22:26:02.459316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.459131 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert\" (UniqueName: \"kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert\") pod \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\" (UID: \"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb\") " Oct 29 22:26:02.461450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.461423 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert" pod "df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" (UID: "df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:26:02.461538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.461487 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r" pod "df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" (UID: "df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb"). InnerVolumeSpecName "kube-api-access-rnz2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:02.560076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.560041 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnz2r\" (UniqueName: \"kubernetes.io/projected/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-kube-api-access-rnz2r\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:26:02.560076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:02.560071 5073 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:26:03.105193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.105096 5073 generic.go:356] "Generic (PLEG): container finished" podID="df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" containerID="a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7" exitCode=0 Oct 29 22:26:03.105193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.105179 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" Oct 29 22:26:03.105736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.105189 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" event={"ID":"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb","Type":"ContainerDied","Data":"a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7"} Oct 29 22:26:03.105736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.105262 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m" event={"ID":"df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb","Type":"ContainerDied","Data":"d2c8358b88e5a28b48349c9d18736383c61d0e09cef18401bfa197412c3c5296"} Oct 29 22:26:03.105736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.105293 5073 scope.go:122] "RemoveContainer" containerID="a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7" Oct 29 22:26:03.114271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.114251 5073 scope.go:122] "RemoveContainer" containerID="a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7" Oct 29 22:26:03.114524 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:26:03.114501 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7\": container with ID starting with a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7 not found: ID does not exist" containerID="a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7" Oct 29 22:26:03.114580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.114537 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7"} err="failed to get container status \"a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7\": rpc error: code = NotFound desc = could not find container \"a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7\": container with ID starting with a32c0b216b5b684fab9f2b870110963fb21991ff111d5e745cce2307e39ab9b7 not found: ID does not exist" Oct 29 22:26:03.159333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.159294 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m"] Oct 29 22:26:03.172896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:03.172866 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m"] Oct 29 22:26:04.836354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:04.836319 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" path="/var/lib/kubelet/pods/df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb/volumes" Oct 29 22:26:09.256651 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:26:09.256608 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:26:22.721470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:22.721439 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:26:26.475175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:26.475123 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:26:31.058346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:31.058316 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:26:35.540863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:35.540828 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:26:39.260208 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:26:39.260141 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:26:40.207451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:26:40.207412 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:27:09.254234 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:27:09.254191 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:27:12.166567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.166536 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s"] Oct 29 22:27:12.166979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.166948 5073 cpu_manager.go:414] "containerMap: removing container" podUID="df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" containerName="manager" Oct 29 22:27:12.166979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.166963 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" containerName="manager" Oct 29 22:27:12.167100 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.167063 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4ce2eb-e9fd-4a21-93cf-0071ff28b9bb" containerName="manager" Oct 29 22:27:12.170080 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.170059 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.180135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.180115 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp"] Oct 29 22:27:12.183583 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.183564 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.196765 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.196742 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s"] Oct 29 22:27:12.204640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.204618 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp"] Oct 29 22:27:12.339303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339277 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339308 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339330 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339434 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339499 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339529 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbkp\" (UniqueName: \"kubernetes.io/projected/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-kube-api-access-jdbkp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339566 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339584 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339702 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339622 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339702 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339675 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339807 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339725 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339807 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339790 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339827 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339860 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339910 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4947t\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339953 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.339991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339970 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.339991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.339984 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.440955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.440894 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.440955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.440921 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.440955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.440940 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.440965 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.440987 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4947t\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441013 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441038 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441062 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441093 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441119 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441181 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441209 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441250 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441271 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbkp\" (UniqueName: \"kubernetes.io/projected/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-kube-api-access-jdbkp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441326 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441351 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441379 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441407 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441436 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441457 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441619 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441850 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.441968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441897 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.441968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441944 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.442065 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.441982 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.442276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.442252 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.442549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.442336 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.442616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.442602 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.443935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.443910 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.444455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.444433 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.444504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.444445 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.444949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.444933 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.462617 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.462587 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.462734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.462713 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbkp\" (UniqueName: \"kubernetes.io/projected/9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a-kube-api-access-jdbkp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp\" (UID: \"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.462777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.462761 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4947t\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.463066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.463050 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.482994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.482976 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:12.498416 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.498392 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:12.808729 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.808706 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s"] Oct 29 22:27:12.810861 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:27:12.810835 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf58a7a_5a11_4f93_a8ba_cd8982811b39.slice/crio-9741dabd74202671f166f9985717fae82cb8ebe7ec262a93395712fdecad5c02 WatchSource:0}: Error finding container 9741dabd74202671f166f9985717fae82cb8ebe7ec262a93395712fdecad5c02: Status 404 returned error can't find the container with id 9741dabd74202671f166f9985717fae82cb8ebe7ec262a93395712fdecad5c02 Oct 29 22:27:12.811324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.811303 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp"] Oct 29 22:27:12.812761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.812734 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:27:12.812843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.812800 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:27:12.812880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.812854 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:27:12.822571 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:27:12.822542 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b09d9c3_9695_41d3_bbb7_e7a4f4f87c9a.slice/crio-17fa89c288605e833d064479436d8064d3c6b96c667aeaff786881c6299863fb WatchSource:0}: Error finding container 17fa89c288605e833d064479436d8064d3c6b96c667aeaff786881c6299863fb: Status 404 returned error can't find the container with id 17fa89c288605e833d064479436d8064d3c6b96c667aeaff786881c6299863fb Oct 29 22:27:12.824849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.824827 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:27:12.824898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.824879 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:27:12.824935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:12.824907 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:27:13.453119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.453080 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" event={"ID":"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a","Type":"ContainerStarted","Data":"0e74661a2ce5aa4b764a2402802a5be9904e33f67af3b48e4191c4324a0ca0d1"} Oct 29 22:27:13.453705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.453681 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" event={"ID":"9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a","Type":"ContainerStarted","Data":"17fa89c288605e833d064479436d8064d3c6b96c667aeaff786881c6299863fb"} Oct 29 22:27:13.455948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.455925 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" event={"ID":"eaf58a7a-5a11-4f93-a8ba-cd8982811b39","Type":"ContainerStarted","Data":"d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf"} Oct 29 22:27:13.456120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.456105 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" event={"ID":"eaf58a7a-5a11-4f93-a8ba-cd8982811b39","Type":"ContainerStarted","Data":"9741dabd74202671f166f9985717fae82cb8ebe7ec262a93395712fdecad5c02"} Oct 29 22:27:13.483435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.483402 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:13.484943 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.484921 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:13.485039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.484975 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:13.498824 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.498801 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:13.499855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.499838 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:13.499932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.499884 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:13.536568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.536505 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podStartSLOduration=1.536485538 podStartE2EDuration="1.536485538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:13.512586229 +0000 UTC m=+748.827980332" watchObservedRunningTime="2026-10-29 22:27:13.536485538 +0000 UTC m=+748.851879621" Oct 29 22:27:13.572612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:13.572564 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podStartSLOduration=1.5725506710000001 podStartE2EDuration="1.572550671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:13.571758213 +0000 UTC m=+748.887152286" watchObservedRunningTime="2026-10-29 22:27:13.572550671 +0000 UTC m=+748.887944750" Oct 29 22:27:14.483701 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:14.483672 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:14.484101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:14.483722 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:14.499043 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:14.499015 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:14.499202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:14.499056 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:15.483982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:15.483949 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:15.484398 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:15.484005 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:15.499309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:15.499280 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:15.499420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:15.499316 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:16.484240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:16.484203 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:16.484753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:16.484263 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:16.499285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:16.499257 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:16.499415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:16.499320 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:17.483659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:17.483625 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:17.483829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:17.483672 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:17.499538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:17.499510 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:17.499822 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:17.499561 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:18.484106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:18.484072 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:18.484306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:18.484133 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:18.499472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:18.499443 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:18.499621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:18.499483 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:19.484038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:19.484003 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:27:19.484263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:19.484058 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 29 22:27:19.499315 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:19.499289 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:27:19.499419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:19.499329 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 29 22:27:21.143265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:21.143232 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:21.144013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:21.143961 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:21.144881 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:21.144849 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:27:21.172353 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:21.172328 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:21.172776 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:21.172752 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:21.173584 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:21.173539 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" Oct 29 22:27:39.258096 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:27:39.258055 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:27:56.031436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:56.031399 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:27:59.628208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.628171 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5"] Oct 29 22:27:59.631986 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.631965 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.663669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.663452 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:27:59.663669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.663458 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:27:59.663669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.663492 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-40cf9a-serving-cert\"" Oct 29 22:27:59.663669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.663585 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 22:27:59.663669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.663646 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:27:59.665542 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.665519 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5"] Oct 29 22:27:59.771812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.771781 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.771965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.771836 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.771965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.771925 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vwm\" (UniqueName: \"kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.772068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.772006 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.873213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.873167 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.873213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.873213 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.873669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.873242 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vwm\" (UniqueName: \"kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.873669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.873310 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.873996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.873976 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.875726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.875704 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.875853 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.875835 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.895982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.895922 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vwm\" (UniqueName: \"kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm\") pod \"maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:27:59.944784 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:27:59.944754 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:28:00.253874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:00.253771 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5"] Oct 29 22:28:00.256305 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:00.256278 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc789c8b7_746a_43e3_b087_0f8b291cb446.slice/crio-8373a0e3ce31baa35bb1e57cdbf0e2c6865e9a10917e92f0e2b1aabf555d0dae WatchSource:0}: Error finding container 8373a0e3ce31baa35bb1e57cdbf0e2c6865e9a10917e92f0e2b1aabf555d0dae: Status 404 returned error can't find the container with id 8373a0e3ce31baa35bb1e57cdbf0e2c6865e9a10917e92f0e2b1aabf555d0dae Oct 29 22:28:00.687903 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:00.687820 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" event={"ID":"c789c8b7-746a-43e3-b087-0f8b291cb446","Type":"ContainerStarted","Data":"8373a0e3ce31baa35bb1e57cdbf0e2c6865e9a10917e92f0e2b1aabf555d0dae"} Oct 29 22:28:03.592637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.592606 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m"] Oct 29 22:28:03.596326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.596307 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.611237 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.611210 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.611343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.611259 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.611437 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.611419 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lcs\" (UniqueName: \"kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.611484 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.611468 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.631168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.626526 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4a0287-serving-cert\"" Oct 29 22:28:03.631168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.628502 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m"] Oct 29 22:28:03.706038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.706004 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" event={"ID":"c789c8b7-746a-43e3-b087-0f8b291cb446","Type":"ContainerStarted","Data":"be6fd702782683b1faa3a1945cfdce8e47c75ace9570e280163bb9a2e25a42eb"} Oct 29 22:28:03.706212 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.706131 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:28:03.712714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.712687 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.712826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.712744 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.712826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.712793 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lcs\" (UniqueName: \"kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.712938 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:28:03.712833 5073 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-serving-cert: secret "maas-api-e2e-worker-w4-4a0287-serving-cert" not found Oct 29 22:28:03.712938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.712841 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.712938 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:28:03.712908 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls podName:4e538166-0db6-4b8b-b304-0543406f5e01 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:04.212887168 +0000 UTC m=+799.528281243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls") pod "maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" (UID: "4e538166-0db6-4b8b-b304-0543406f5e01") : secret "maas-api-e2e-worker-w4-4a0287-serving-cert" not found Oct 29 22:28:03.713505 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.713483 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.715414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.715399 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.735487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.735459 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lcs\" (UniqueName: \"kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:03.752420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:03.752373 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" podStartSLOduration=2.377040817 podStartE2EDuration="4.752362053s" totalImagesPullingTime="2.375321236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:03.752173657 +0000 UTC m=+799.067567738" watchObservedRunningTime="2026-10-29 22:28:03.752362053 +0000 UTC m=+799.067756133" Oct 29 22:28:04.217555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:04.217522 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:04.220136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:04.220108 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:04.508745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:04.508663 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:04.630229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:04.630195 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:28:04.806899 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:04.806877 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m"] Oct 29 22:28:04.808748 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:04.808725 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e538166_0db6_4b8b_b304_0543406f5e01.slice/crio-0e3e777a2f5f69efcbb024c45e081ff09d51d53cb2c951e594c44b0b63d51126 WatchSource:0}: Error finding container 0e3e777a2f5f69efcbb024c45e081ff09d51d53cb2c951e594c44b0b63d51126: Status 404 returned error can't find the container with id 0e3e777a2f5f69efcbb024c45e081ff09d51d53cb2c951e594c44b0b63d51126 Oct 29 22:28:05.720186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:05.720131 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" event={"ID":"4e538166-0db6-4b8b-b304-0543406f5e01","Type":"ContainerStarted","Data":"2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623"} Oct 29 22:28:05.720552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:05.720202 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" event={"ID":"4e538166-0db6-4b8b-b304-0543406f5e01","Type":"ContainerStarted","Data":"0e3e777a2f5f69efcbb024c45e081ff09d51d53cb2c951e594c44b0b63d51126"} Oct 29 22:28:05.720552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:05.720224 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:05.760882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:05.760834 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" podStartSLOduration=2.460406946 podStartE2EDuration="2.760823163s" totalImagesPullingTime="300.416217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:05.76069064 +0000 UTC m=+801.076084720" watchObservedRunningTime="2026-10-29 22:28:05.760823163 +0000 UTC m=+801.076217243" Oct 29 22:28:09.253701 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:28:09.253661 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:28:09.719313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:09.719289 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 22:28:10.839088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:10.839061 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:28:11.730024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:11.729997 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 22:28:18.302958 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.302926 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6"] Oct 29 22:28:18.307591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.307574 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341372 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341425 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341454 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341636 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341601 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341742 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341660 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341764 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341819 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njd6f\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341909 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.341959 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.341941 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.347344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.347319 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6"] Oct 29 22:28:18.383722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.383695 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0d12c5-openshift-default-dockercfg-477sq\"" Oct 29 22:28:18.442900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.442869 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.442922 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njd6f\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.442943 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.442962 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443014 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443051 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443075 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443129 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443334 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443431 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.443915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443888 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.444032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.443952 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.444307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.444281 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.444484 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.444300 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.446596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.446571 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.446801 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.446775 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.470984 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.470955 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.476178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.476120 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njd6f\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.622881 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.622796 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:18.997371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:18.997336 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6"] Oct 29 22:28:19.003356 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:19.003326 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55317bea_e9ea_4d29_ad8f_4996674bd8fe.slice/crio-496ec90c62effe12858c2d4414b2dcf4089ee5612de7ece4b3cf72e85ca0dc71 WatchSource:0}: Error finding container 496ec90c62effe12858c2d4414b2dcf4089ee5612de7ece4b3cf72e85ca0dc71: Status 404 returned error can't find the container with id 496ec90c62effe12858c2d4414b2dcf4089ee5612de7ece4b3cf72e85ca0dc71 Oct 29 22:28:19.005629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:19.005597 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:19.005722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:19.005658 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:19.005722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:19.005683 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:19.792013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:19.791974 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" event={"ID":"55317bea-e9ea-4d29-ad8f-4996674bd8fe","Type":"ContainerStarted","Data":"ecc75e5b1caaa78d73d8c23162042b51d0921bf9c0739b1a4096a04a88e5320f"} Oct 29 22:28:19.792452 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:19.792032 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" event={"ID":"55317bea-e9ea-4d29-ad8f-4996674bd8fe","Type":"ContainerStarted","Data":"496ec90c62effe12858c2d4414b2dcf4089ee5612de7ece4b3cf72e85ca0dc71"} Oct 29 22:28:19.852569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:19.852520 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podStartSLOduration=1.85250577 podStartE2EDuration="1.85250577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:19.851659948 +0000 UTC m=+815.167054027" watchObservedRunningTime="2026-10-29 22:28:19.85250577 +0000 UTC m=+815.167899849" Oct 29 22:28:20.625176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:20.623635 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:20.639390 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:20.639353 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:20.639594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:20.639416 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:28:21.633279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:21.633245 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:21.633889 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:21.633835 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:28:22.068712 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:22.068672 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:28:22.626526 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:22.626494 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:22.626790 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:22.626762 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:28:23.624581 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:23.624544 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:23.625225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:23.625192 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:28:24.623480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:24.623446 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:28:24.623751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:24.623725 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:28:25.627740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:25.627710 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:25.628199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:25.628004 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:25.628695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:25.628678 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:28:31.140384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.140352 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:28:31.174571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.174540 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx"] Oct 29 22:28:31.178827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.178805 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.200737 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.200711 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx"] Oct 29 22:28:31.212261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.212236 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"kube-root-ca.crt\"" Oct 29 22:28:31.212398 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.212335 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"simulator-w0-40cf9a-kserve-self-signed-certs\"" Oct 29 22:28:31.212398 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.212388 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"openshift-service-ca.crt\"" Oct 29 22:28:31.212504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.212457 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"default-dockercfg-b5wbl\"" Oct 29 22:28:31.290612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.290575 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.290792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.290728 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tzk\" (UniqueName: \"kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.290864 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.290845 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.290930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.290913 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.290997 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.290982 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.291071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.291052 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392075 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392120 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tzk\" (UniqueName: \"kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392191 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392219 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392265 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392291 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392617 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392736 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.392816 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.392791 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.395042 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.395008 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.395195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.395076 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.416943 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.416911 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tzk\" (UniqueName: \"kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk\") pod \"simulator-w0-40cf9a-kserve-589ff674d6-cqdxx\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.503737 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.503709 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:31.804770 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.804745 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx"] Oct 29 22:28:31.806463 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:31.806434 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fd2b64d_24fa_4572_adb7_bfc7ebac61d3.slice/crio-4bbc86061c211551958624521f829a139d8e952405f16e68aa15500534a1c252 WatchSource:0}: Error finding container 4bbc86061c211551958624521f829a139d8e952405f16e68aa15500534a1c252: Status 404 returned error can't find the container with id 4bbc86061c211551958624521f829a139d8e952405f16e68aa15500534a1c252 Oct 29 22:28:31.866119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:31.866088 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" event={"ID":"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3","Type":"ContainerStarted","Data":"4bbc86061c211551958624521f829a139d8e952405f16e68aa15500534a1c252"} Oct 29 22:28:33.880247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:33.880184 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" event={"ID":"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3","Type":"ContainerStarted","Data":"e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a"} Oct 29 22:28:33.925301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:33.925255 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podStartSLOduration=1.859017103 podStartE2EDuration="2.925241653s" totalImagesPullingTime="1.06622455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:33.924415724 +0000 UTC m=+829.239809806" watchObservedRunningTime="2026-10-29 22:28:33.925241653 +0000 UTC m=+829.240635732" Oct 29 22:28:34.504027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:34.503995 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:37.261978 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.261946 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl"] Oct 29 22:28:37.269122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.269099 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.290295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.290259 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"kube-root-ca.crt\"" Oct 29 22:28:37.290949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.290928 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl"] Oct 29 22:28:37.302203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.302178 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"default-dockercfg-f6qvf\"" Oct 29 22:28:37.302335 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.302267 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"simulator-w6-92fd86-kserve-self-signed-certs\"" Oct 29 22:28:37.348048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.348023 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.348167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.348072 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.348167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.348096 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtxlp\" (UniqueName: \"kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.348242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.348195 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.348276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.348250 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.348360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.348348 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.394987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.394968 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"openshift-service-ca.crt\"" Oct 29 22:28:37.449316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449287 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.449453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449359 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.449453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449405 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.449453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtxlp\" (UniqueName: \"kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.449619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449474 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.449619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449538 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.450021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449739 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.450021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.449905 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.450214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.450085 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.452594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.452570 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.452867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.452848 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.472363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.472338 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtxlp\" (UniqueName: \"kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp\") pod \"simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.585037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.583666 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:37.888127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.888101 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl"] Oct 29 22:28:37.889788 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:37.889758 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e44bd4e_871d_48b2_9c8d_ea4a692f9ff7.slice/crio-1a3ad88f09bb73594d7526c5f7c556a91e89de7a05ac5346f12b94b31c8e2d7e WatchSource:0}: Error finding container 1a3ad88f09bb73594d7526c5f7c556a91e89de7a05ac5346f12b94b31c8e2d7e: Status 404 returned error can't find the container with id 1a3ad88f09bb73594d7526c5f7c556a91e89de7a05ac5346f12b94b31c8e2d7e Oct 29 22:28:37.901708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:37.901675 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" event={"ID":"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7","Type":"ContainerStarted","Data":"1a3ad88f09bb73594d7526c5f7c556a91e89de7a05ac5346f12b94b31c8e2d7e"} Oct 29 22:28:38.907975 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:38.907944 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" event={"ID":"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7","Type":"ContainerStarted","Data":"03906e1f8c098c0777bd269557010aebe9ac266aa5cef02740cae6532556c41b"} Oct 29 22:28:38.950519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:38.950469 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podStartSLOduration=1.950457092 podStartE2EDuration="1.950457092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:38.950020097 +0000 UTC m=+834.265414195" watchObservedRunningTime="2026-10-29 22:28:38.950457092 +0000 UTC m=+834.265851172" Oct 29 22:28:39.254999 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:28:39.254960 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:28:39.584811 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:39.584728 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:41.504070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:41.504035 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:41.516271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:41.516246 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:41.934213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:41.934116 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:28:42.202787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:42.202707 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:28:47.149914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.149884 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb"] Oct 29 22:28:47.155445 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.155419 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.171253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.171227 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb"] Oct 29 22:28:47.183024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.182992 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"simulator-w5-0d12c5-kserve-self-signed-certs\"" Oct 29 22:28:47.185615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.185592 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"default-dockercfg-9x74l\"" Oct 29 22:28:47.185699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.185645 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"kube-root-ca.crt\"" Oct 29 22:28:47.185699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.185677 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"openshift-service-ca.crt\"" Oct 29 22:28:47.339443 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.339403 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.339619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.339453 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.339619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.339532 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.339619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.339587 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.339619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.339613 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.339778 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.339738 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zktg7\" (UniqueName: \"kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440405 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440435 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440451 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440468 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440484 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440516 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zktg7\" (UniqueName: \"kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440877 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.440971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440911 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.441012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.440966 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.443056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.443024 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.443371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.443354 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.463943 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.463916 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zktg7\" (UniqueName: \"kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7\") pod \"simulator-w5-0d12c5-kserve-585b9fcf57-872pb\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.472805 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.472778 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:47.584685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.584651 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:47.598163 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.598128 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:47.783520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.783497 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb"] Oct 29 22:28:47.784629 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:47.784604 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0da1f6f_33f4_411f_806e_6e1e37fc03f6.slice/crio-1a2f0f4f929d58c5b30f91f9407a4febf15bf5a26360620d72bf9ab12a420e22 WatchSource:0}: Error finding container 1a2f0f4f929d58c5b30f91f9407a4febf15bf5a26360620d72bf9ab12a420e22: Status 404 returned error can't find the container with id 1a2f0f4f929d58c5b30f91f9407a4febf15bf5a26360620d72bf9ab12a420e22 Oct 29 22:28:47.954379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.954345 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" event={"ID":"a0da1f6f-33f4-411f-806e-6e1e37fc03f6","Type":"ContainerStarted","Data":"4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96"} Oct 29 22:28:47.954551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.954387 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" event={"ID":"a0da1f6f-33f4-411f-806e-6e1e37fc03f6","Type":"ContainerStarted","Data":"1a2f0f4f929d58c5b30f91f9407a4febf15bf5a26360620d72bf9ab12a420e22"} Oct 29 22:28:47.965744 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.965719 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:28:47.996425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:47.996385 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podStartSLOduration=0.99637404 podStartE2EDuration="996.37404ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:47.994967861 +0000 UTC m=+843.310361940" watchObservedRunningTime="2026-10-29 22:28:47.99637404 +0000 UTC m=+843.311768119" Oct 29 22:28:48.473843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.473813 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:48.519210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.519182 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85"] Oct 29 22:28:48.523087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.523061 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.545962 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.545936 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-95672c-openshift-default-dockercfg-hp48t\"" Oct 29 22:28:48.548513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.548486 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85"] Oct 29 22:28:48.651451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651425 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvp45\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651468 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651487 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651541 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651673 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651648 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651675 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651698 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651785 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.651837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.651820 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752547 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752574 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752623 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp45\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752650 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752668 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752683 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752705 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752723 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.752771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752751 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.753116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.752972 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.753116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.753011 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.753286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.753265 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.753356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.753341 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.753497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.753480 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.755250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.755226 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.755354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.755320 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.773526 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.773493 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp45\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.773623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.773543 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:48.836364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:48.836341 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:49.163924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:49.163898 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85"] Oct 29 22:28:49.165362 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:49.165331 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd216984e_69e8_445d_b3e0_334ca271c0fd.slice/crio-1130e01db25c4b6bc5ab51d47d072f16b60594ed9a88c972ac213b53a6f55349 WatchSource:0}: Error finding container 1130e01db25c4b6bc5ab51d47d072f16b60594ed9a88c972ac213b53a6f55349: Status 404 returned error can't find the container with id 1130e01db25c4b6bc5ab51d47d072f16b60594ed9a88c972ac213b53a6f55349 Oct 29 22:28:49.167746 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:49.167710 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:49.167833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:49.167779 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:49.167833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:49.167805 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:49.966443 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:49.966405 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" event={"ID":"d216984e-69e8-445d-b3e0-334ca271c0fd","Type":"ContainerStarted","Data":"c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51"} Oct 29 22:28:49.966950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:49.966465 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" event={"ID":"d216984e-69e8-445d-b3e0-334ca271c0fd","Type":"ContainerStarted","Data":"1130e01db25c4b6bc5ab51d47d072f16b60594ed9a88c972ac213b53a6f55349"} Oct 29 22:28:50.030046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:50.029977 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podStartSLOduration=2.029956318 podStartE2EDuration="2.029956318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:50.02549085 +0000 UTC m=+845.340884931" watchObservedRunningTime="2026-10-29 22:28:50.029956318 +0000 UTC m=+845.345350400" Oct 29 22:28:50.838934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:50.838886 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:28:50.839134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:50.838957 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:28:50.842555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:50.842531 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:51.837610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:51.837577 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:28:51.838083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:51.837633 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:28:51.863624 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:51.863587 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:28:52.838566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:52.838533 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:28:52.839169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:52.839119 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:28:53.447137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.447102 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w"] Oct 29 22:28:53.452552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.452526 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.476193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.476165 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w"] Oct 29 22:28:53.483738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.483705 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"openshift-service-ca.crt\"" Oct 29 22:28:53.483874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.483765 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"kube-root-ca.crt\"" Oct 29 22:28:53.483951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.483903 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"default-dockercfg-tlzc2\"" Oct 29 22:28:53.484550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.484528 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"premium-w2-8d3632-kserve-self-signed-certs\"" Oct 29 22:28:53.604727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.604692 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.604999 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.604982 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.605236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.605216 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pmw\" (UniqueName: \"kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.605410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.605393 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.605594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.605576 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.605771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.605758 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.707316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.707286 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.707503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.707334 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.707503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.707390 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pmw\" (UniqueName: \"kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.707503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.707413 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.707503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.707454 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.707701 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.707528 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.711181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.708877 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.711181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.709411 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.711181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.709641 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.714393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.714370 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.735182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.732941 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pmw\" (UniqueName: \"kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.743179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.742641 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm\") pod \"premium-w2-8d3632-kserve-8997cb4-7cl8w\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.769311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.769288 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:53.837077 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.837043 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:28:53.837251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:53.837090 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:28:54.089740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:54.089717 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w"] Oct 29 22:28:54.092724 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:28:54.092696 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod053148a5_5ce2_4131_9aab_c09455de26b4.slice/crio-e5519f851c7f4899c81a6c716b6ae03bf82afb22e4c41c9defb0f65b8279524f WatchSource:0}: Error finding container e5519f851c7f4899c81a6c716b6ae03bf82afb22e4c41c9defb0f65b8279524f: Status 404 returned error can't find the container with id e5519f851c7f4899c81a6c716b6ae03bf82afb22e4c41c9defb0f65b8279524f Oct 29 22:28:54.837523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:54.837477 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:28:54.837808 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:54.837768 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:28:54.996036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:54.995987 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" event={"ID":"053148a5-5ce2-4131-9aab-c09455de26b4","Type":"ContainerStarted","Data":"877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19"} Oct 29 22:28:54.996264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:54.996049 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" event={"ID":"053148a5-5ce2-4131-9aab-c09455de26b4","Type":"ContainerStarted","Data":"e5519f851c7f4899c81a6c716b6ae03bf82afb22e4c41c9defb0f65b8279524f"} Oct 29 22:28:55.036875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:55.036833 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podStartSLOduration=2.036820742 podStartE2EDuration="2.036820742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:55.036367361 +0000 UTC m=+850.351761443" watchObservedRunningTime="2026-10-29 22:28:55.036820742 +0000 UTC m=+850.352214819" Oct 29 22:28:55.770269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:55.770233 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:28:55.841675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:55.841648 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:55.842101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:55.842085 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:55.842682 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:55.842668 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:28:57.474419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:57.474386 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:57.486877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:57.486852 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:28:58.023260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:28:58.023235 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:29:02.622975 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.622937 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l"] Oct 29 22:29:02.627791 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.627765 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.644409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.644373 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"openshift-service-ca.crt\"" Oct 29 22:29:02.644409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.644373 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"kube-root-ca.crt\"" Oct 29 22:29:02.644800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.644778 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l"] Oct 29 22:29:02.655234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.655203 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"default-dockercfg-t54gx\"" Oct 29 22:29:02.655234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.655216 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"premium-w3-95672c-kserve-self-signed-certs\"" Oct 29 22:29:02.722792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.722755 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.723001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.722833 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.723075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.722993 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.723075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.723033 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7bm\" (UniqueName: \"kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.723198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.723173 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.723253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.723233 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.824912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.824867 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.825216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.825192 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.825332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.825318 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.825402 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.825381 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.825461 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.825418 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7bm\" (UniqueName: \"kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.825511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.825482 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.826259 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.826227 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.826412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.826308 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.826560 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.826535 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.829613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.829582 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.829984 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.829932 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.850184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.850117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7bm\" (UniqueName: \"kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm\") pod \"premium-w3-95672c-kserve-558d9c9f65-9927l\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:02.941871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:02.941772 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:03.243821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.243789 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l"] Oct 29 22:29:03.245051 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:29:03.245024 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde5beee_062c_4a4a_9993_998e693ffe17.slice/crio-8719ba17a0860d79d144ef92d538c329ad9ea49724d607d4ce2ccd1f9f96af5c WatchSource:0}: Error finding container 8719ba17a0860d79d144ef92d538c329ad9ea49724d607d4ce2ccd1f9f96af5c: Status 404 returned error can't find the container with id 8719ba17a0860d79d144ef92d538c329ad9ea49724d607d4ce2ccd1f9f96af5c Oct 29 22:29:03.770126 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.770091 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:29:03.782818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.782795 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:29:03.937661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.937630 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz"] Oct 29 22:29:03.943321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.943296 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:03.961697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.961668 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz"] Oct 29 22:29:03.961999 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.961982 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"openshift-service-ca.crt\"" Oct 29 22:29:03.962106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.962047 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"kube-root-ca.crt\"" Oct 29 22:29:03.962369 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.962347 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"default-dockercfg-jks2w\"" Oct 29 22:29:03.972828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:03.972807 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-879596\"/\"premium-w1-879596-kserve-self-signed-certs\"" Oct 29 22:29:04.039830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.039751 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.039830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.039789 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.039830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.039813 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.040188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.039911 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.040188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.039944 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7gk\" (UniqueName: \"kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.040188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.040082 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.047181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.047117 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" event={"ID":"fde5beee-062c-4a4a-9993-998e693ffe17","Type":"ContainerStarted","Data":"fde199883627442e34b2807f229e49b3e5beb22562cca96723ccf8880929707f"} Oct 29 22:29:04.047944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.047922 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" event={"ID":"fde5beee-062c-4a4a-9993-998e693ffe17","Type":"ContainerStarted","Data":"8719ba17a0860d79d144ef92d538c329ad9ea49724d607d4ce2ccd1f9f96af5c"} Oct 29 22:29:04.066772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.066744 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:29:04.091221 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.091133 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podStartSLOduration=2.091121102 podStartE2EDuration="2.091121102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:04.090468799 +0000 UTC m=+859.405862881" watchObservedRunningTime="2026-10-29 22:29:04.091121102 +0000 UTC m=+859.406515183" Oct 29 22:29:04.141658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.141620 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.141833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.141696 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.141833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.141733 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.141833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.141768 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.141833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.141806 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gk\" (UniqueName: \"kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.142068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.141926 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.142121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.142094 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.142982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.142612 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.143410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.143388 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.146515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.146493 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.146817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.146795 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.163757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.163729 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7gk\" (UniqueName: \"kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk\") pod \"premium-w1-879596-kserve-6778b6669c-jj5fz\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.256475 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.256443 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:04.565447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.565418 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz"] Oct 29 22:29:04.568214 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:29:04.568130 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab8ffa39_241f_4f94_9d5b_4ecdbecee27a.slice/crio-40334f135d9478014872fa49d3befa85442a8b3c6f05724a7a7ce43be7ee3b2a WatchSource:0}: Error finding container 40334f135d9478014872fa49d3befa85442a8b3c6f05724a7a7ce43be7ee3b2a: Status 404 returned error can't find the container with id 40334f135d9478014872fa49d3befa85442a8b3c6f05724a7a7ce43be7ee3b2a Oct 29 22:29:04.942019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:04.941938 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:05.053333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:05.053296 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" event={"ID":"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a","Type":"ContainerStarted","Data":"e0812e81e71c0252117cc625d6bc72ba700455833f24fb398d15ac4a08ef9a48"} Oct 29 22:29:05.053333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:05.053337 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" event={"ID":"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a","Type":"ContainerStarted","Data":"40334f135d9478014872fa49d3befa85442a8b3c6f05724a7a7ce43be7ee3b2a"} Oct 29 22:29:05.102338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:05.102290 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podStartSLOduration=2.102275965 podStartE2EDuration="2.102275965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:05.101297423 +0000 UTC m=+860.416691502" watchObservedRunningTime="2026-10-29 22:29:05.102275965 +0000 UTC m=+860.417670101" Oct 29 22:29:05.256932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:05.256895 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:06.826103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:06.826061 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:29:09.254754 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:29:09.254709 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:29:12.942804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:12.942764 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:12.955765 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:12.955723 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:13.110239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:13.110209 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:29:14.257452 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:14.257406 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:14.269837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:14.269812 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:15.121290 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:15.121257 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:29:18.870666 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:18.870633 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj"] Oct 29 22:29:18.876806 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:18.876784 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:18.902952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:18.902920 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj"] Oct 29 22:29:18.912009 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:18.911986 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"premium-w6-92fd86-kserve-self-signed-certs\"" Oct 29 22:29:19.001076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.001041 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.001291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.001110 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.001291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.001269 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.001412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.001334 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.001412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.001390 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4shg\" (UniqueName: \"kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.001587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.001562 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.102831 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.102911 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.102947 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.102979 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.103020 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4shg\" (UniqueName: \"kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.103115 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.103631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.103864 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.104098 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.108182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.108090 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.116859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.115403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.131018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.130942 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4shg\" (UniqueName: \"kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg\") pod \"premium-w6-92fd86-kserve-69cf98bf6f-hhtkj\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.191099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.191062 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:19.503263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:19.503234 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj"] Oct 29 22:29:19.505183 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:29:19.505134 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod734b0726_6df3_4de9_b57d_1dfc752b01de.slice/crio-cbda20ef7aef547736a344b4d1ca297819bff264a2fe13fea87dd8ff08dcdccf WatchSource:0}: Error finding container cbda20ef7aef547736a344b4d1ca297819bff264a2fe13fea87dd8ff08dcdccf: Status 404 returned error can't find the container with id cbda20ef7aef547736a344b4d1ca297819bff264a2fe13fea87dd8ff08dcdccf Oct 29 22:29:20.141769 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:20.141732 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" event={"ID":"734b0726-6df3-4de9-b57d-1dfc752b01de","Type":"ContainerStarted","Data":"5056e022f3bc80c82830dcf9ee1c8b503add1d2843abcd89133344e3ef7029fc"} Oct 29 22:29:20.141769 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:20.141782 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" event={"ID":"734b0726-6df3-4de9-b57d-1dfc752b01de","Type":"ContainerStarted","Data":"cbda20ef7aef547736a344b4d1ca297819bff264a2fe13fea87dd8ff08dcdccf"} Oct 29 22:29:20.189480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:20.189427 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podStartSLOduration=2.189413835 podStartE2EDuration="2.189413835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:20.188096445 +0000 UTC m=+875.503490524" watchObservedRunningTime="2026-10-29 22:29:20.189413835 +0000 UTC m=+875.504807915" Oct 29 22:29:20.191171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:20.191114 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:25.209836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.209796 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:29:25.455798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.455764 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt"] Oct 29 22:29:25.460981 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.460914 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.491829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.491800 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"openshift-service-ca.crt\"" Oct 29 22:29:25.491829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.491812 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"default-dockercfg-zfq69\"" Oct 29 22:29:25.492008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.491851 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"kube-root-ca.crt\"" Oct 29 22:29:25.492008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.491889 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"premium-w4-4a0287-kserve-self-signed-certs\"" Oct 29 22:29:25.494089 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.494067 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt"] Oct 29 22:29:25.566567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.566543 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.566670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.566657 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.566719 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.566699 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.566789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.566777 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.566827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.566803 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsdt\" (UniqueName: \"kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.566827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.566823 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.668843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.668799 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.669050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.668873 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.669050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.668975 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.669050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.669024 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsdt\" (UniqueName: \"kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.669302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.669057 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.669302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.669115 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.669638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.669614 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.672635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.672548 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.675870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.675845 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.676504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.676461 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.676745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.676728 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.705677 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.705645 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsdt\" (UniqueName: \"kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt\") pod \"premium-w4-4a0287-kserve-686596b5f4-cf6kt\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:25.775442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:25.775358 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:26.086995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:26.086909 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt"] Oct 29 22:29:26.090093 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:29:26.090064 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a81ab41_82ec_4eb1_852f_c8d5ce53fa79.slice/crio-dce93024aa016d83e6656bec2558e0a12d74238722efc697a83699988e4bc811 WatchSource:0}: Error finding container dce93024aa016d83e6656bec2558e0a12d74238722efc697a83699988e4bc811: Status 404 returned error can't find the container with id dce93024aa016d83e6656bec2558e0a12d74238722efc697a83699988e4bc811 Oct 29 22:29:26.185071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:26.185001 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" event={"ID":"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79","Type":"ContainerStarted","Data":"dce93024aa016d83e6656bec2558e0a12d74238722efc697a83699988e4bc811"} Oct 29 22:29:27.191771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:27.191737 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" event={"ID":"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79","Type":"ContainerStarted","Data":"0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90"} Oct 29 22:29:27.237514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:27.237465 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podStartSLOduration=2.23744977 podStartE2EDuration="2.23744977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:27.235888379 +0000 UTC m=+882.551282459" watchObservedRunningTime="2026-10-29 22:29:27.23744977 +0000 UTC m=+882.552843851" Oct 29 22:29:27.775875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:27.775842 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:29.191331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:29.191303 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:29.204924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:29.204902 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:29.216593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:29.216574 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:29:33.365557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.365524 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2"] Oct 29 22:29:33.369497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.369473 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.390400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.390379 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-879596-openshift-default-dockercfg-gkrg6\"" Oct 29 22:29:33.406809 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.406783 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2"] Oct 29 22:29:33.552886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.552859 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.552889 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.552906 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.552923 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.553024 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.553060 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.553115 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.553237 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgb5f\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.553285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.553263 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.654758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.654700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.654758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.654755 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.654942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.654879 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.654942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.654917 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655043 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655063 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655093 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655095 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655127 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655188 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655223 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655315 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgb5f\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655318 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.655819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.655769 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.657444 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.657423 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.657761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.657740 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.675708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.675683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.675919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.675900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgb5f\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.683618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.683601 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:33.999193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:33.999145 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2"] Oct 29 22:29:33.999885 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:29:33.999855 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093fcca4_bf99_43c2_83ed_a757047f52e0.slice/crio-abb7c468f52604ac331c5ee53befb79c2bc105f35fcdaa08b68ea822c82b58cb WatchSource:0}: Error finding container abb7c468f52604ac331c5ee53befb79c2bc105f35fcdaa08b68ea822c82b58cb: Status 404 returned error can't find the container with id abb7c468f52604ac331c5ee53befb79c2bc105f35fcdaa08b68ea822c82b58cb Oct 29 22:29:34.002477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.002433 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:29:34.002582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.002516 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:29:34.002582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.002566 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:29:34.232632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.232597 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" event={"ID":"093fcca4-bf99-43c2-83ed-a757047f52e0","Type":"ContainerStarted","Data":"8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38"} Oct 29 22:29:34.232806 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.232655 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" event={"ID":"093fcca4-bf99-43c2-83ed-a757047f52e0","Type":"ContainerStarted","Data":"abb7c468f52604ac331c5ee53befb79c2bc105f35fcdaa08b68ea822c82b58cb"} Oct 29 22:29:34.282019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.281925 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podStartSLOduration=1.2819077380000001 podStartE2EDuration="1.281907738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:34.279298317 +0000 UTC m=+889.594692403" watchObservedRunningTime="2026-10-29 22:29:34.281907738 +0000 UTC m=+889.597301819" Oct 29 22:29:34.684539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:34.684470 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:35.686642 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:35.686606 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:35.687106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:35.686663 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 29 22:29:35.776651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:35.776345 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:35.796935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:35.796905 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:36.272826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:36.272787 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:29:36.684551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:36.684516 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:36.684810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:36.684782 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 29 22:29:37.684404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:37.684370 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:37.684808 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:37.684435 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 29 22:29:38.684330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:38.684301 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:38.684501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:38.684350 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 29 22:29:39.273674 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:29:39.273634 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:29:39.685066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:39.684987 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:39.685066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:39.685048 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 29 22:29:40.684614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:40.684580 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:40.684773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:40.684651 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 29 22:29:41.862119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:41.862088 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:41.862596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:41.862391 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:41.864231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:41.864212 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:29:44.207756 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:44.207723 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:29:44.931493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:44.931460 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:29:44.936616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:44.936587 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:29:44.938900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:44.938873 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:29:44.942432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:29:44.942404 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:30:00.202305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.202263 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj"] Oct 29 22:30:00.217455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.217430 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.244103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.244076 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:30:00.438177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.435369 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4"] Oct 29 22:30:00.488049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.470483 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxln\" (UniqueName: \"kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.488049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.470578 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.504965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.501980 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.533708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.506125 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj"] Oct 29 22:30:00.591174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.591112 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqxf\" (UniqueName: \"kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.597592 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.597552 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxln\" (UniqueName: \"kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.598292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.598268 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.598637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.598616 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.600134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.600083 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.603562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.600561 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.607094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.607075 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj"] Oct 29 22:30:00.625251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.625229 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4"] Oct 29 22:30:00.630938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.630918 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj"] Oct 29 22:30:00.691185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.690474 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxln\" (UniqueName: \"kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.700404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.700375 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.700504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.700424 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-4kznj\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.700504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.700471 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqxf\" (UniqueName: \"kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.700603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.700584 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zqb\" (UniqueName: \"kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb\") pod \"maas-api-key-cleanup-29888550-4kznj\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.701024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.701006 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.776298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.769800 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqxf\" (UniqueName: \"kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.801383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.801359 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zqb\" (UniqueName: \"kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb\") pod \"maas-api-key-cleanup-29888550-4kznj\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.801510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.801458 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-4kznj\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.802048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.802035 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:00.802087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.802065 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-4kznj\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.842223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.828482 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zqb\" (UniqueName: \"kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb\") pod \"maas-api-key-cleanup-29888550-4kznj\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:00.936265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.936239 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:00.979331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:00.979300 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:01.530632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:01.530607 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4"] Oct 29 22:30:01.531569 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:01.531539 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681e0d8b_112f_4de4_aea4_1cff02bd87f6.slice/crio-62523c111a028473416901370b00dd45f19c6ddc887770deb8019cb40742734b WatchSource:0}: Error finding container 62523c111a028473416901370b00dd45f19c6ddc887770deb8019cb40742734b: Status 404 returned error can't find the container with id 62523c111a028473416901370b00dd45f19c6ddc887770deb8019cb40742734b Oct 29 22:30:01.554528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:01.554505 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj"] Oct 29 22:30:01.555275 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:01.555252 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6084d17_c3cb_405b_9c17_d628eec47d49.slice/crio-d1339fdf1749f3038cc1b4a2592a688f6341eef9d98b594beab11a1d833da442 WatchSource:0}: Error finding container d1339fdf1749f3038cc1b4a2592a688f6341eef9d98b594beab11a1d833da442: Status 404 returned error can't find the container with id d1339fdf1749f3038cc1b4a2592a688f6341eef9d98b594beab11a1d833da442 Oct 29 22:30:01.562299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:01.562270 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" event={"ID":"681e0d8b-112f-4de4-aea4-1cff02bd87f6","Type":"ContainerStarted","Data":"62523c111a028473416901370b00dd45f19c6ddc887770deb8019cb40742734b"} Oct 29 22:30:01.563640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:01.563618 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" event={"ID":"a6084d17-c3cb-405b-9c17-d628eec47d49","Type":"ContainerStarted","Data":"d1339fdf1749f3038cc1b4a2592a688f6341eef9d98b594beab11a1d833da442"} Oct 29 22:30:01.653792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:01.653757 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj"] Oct 29 22:30:01.655718 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:01.655689 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af72fb4_d271_4ab1_b369_ed035220d463.slice/crio-df826c655f2ed557510c60a43b396635b1a090962ecfdadbef30740bbf0350d5 WatchSource:0}: Error finding container df826c655f2ed557510c60a43b396635b1a090962ecfdadbef30740bbf0350d5: Status 404 returned error can't find the container with id df826c655f2ed557510c60a43b396635b1a090962ecfdadbef30740bbf0350d5 Oct 29 22:30:02.342914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.342882 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h"] Oct 29 22:30:02.347549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.347530 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.365292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.365204 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:02.381416 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.381387 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h"] Oct 29 22:30:02.424693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.424655 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.424852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.424756 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.424945 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.424917 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrnj\" (UniqueName: \"kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.528037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.526552 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrnj\" (UniqueName: \"kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.528037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.526685 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.528037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.526762 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.528037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.527991 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.529932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.529909 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.551287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.551229 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrnj\" (UniqueName: \"kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj\") pod \"maas-api-revoke-sub-ba896a18-2r24h\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.574920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.574836 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" event={"ID":"9af72fb4-d271-4ab1-b369-ed035220d463","Type":"ContainerStarted","Data":"df826c655f2ed557510c60a43b396635b1a090962ecfdadbef30740bbf0350d5"} Oct 29 22:30:02.664107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.664021 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:02.995696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:02.995662 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h"] Oct 29 22:30:03.002130 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:03.002100 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3344d5cb_ce63_441b_a72f_a34dc68b0e01.slice/crio-c057dd3507cbcdd82d5c1b546280af236cb698e88aa0e05e2d05283dc72d1a0f WatchSource:0}: Error finding container c057dd3507cbcdd82d5c1b546280af236cb698e88aa0e05e2d05283dc72d1a0f: Status 404 returned error can't find the container with id c057dd3507cbcdd82d5c1b546280af236cb698e88aa0e05e2d05283dc72d1a0f Oct 29 22:30:03.162649 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.162614 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4"] Oct 29 22:30:03.168107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.168083 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.189400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.189373 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-40cf9a-openshift-default-dockercfg-8bd6l\"" Oct 29 22:30:03.209724 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.209696 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4"] Oct 29 22:30:03.238965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.238931 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.238982 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239020 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239061 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239100 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239116 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2f98\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239141 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239188 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.239292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.239234 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340322 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340388 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340449 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340507 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340534 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2f98\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340573 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340600 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340643 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340704 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.340915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.340870 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.341109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.341078 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.341315 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.341293 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.341315 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.341312 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.342193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.342130 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.343512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.343482 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.344725 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.344705 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.365940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.365914 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.366102 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.366074 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2f98\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.486861 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.486822 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:03.587367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.587275 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" event={"ID":"3344d5cb-ce63-441b-a72f-a34dc68b0e01","Type":"ContainerStarted","Data":"c057dd3507cbcdd82d5c1b546280af236cb698e88aa0e05e2d05283dc72d1a0f"} Oct 29 22:30:03.845675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.845643 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4"] Oct 29 22:30:03.871011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.870790 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:03.871139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.871063 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:03.871139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:03.871097 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:04.639196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:04.629751 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" event={"ID":"b271344f-e086-4aca-a1d8-8294ea12d756","Type":"ContainerStarted","Data":"b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a"} Oct 29 22:30:04.639196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:04.629876 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" event={"ID":"b271344f-e086-4aca-a1d8-8294ea12d756","Type":"ContainerStarted","Data":"bb76927ccd2581a2b46c835bbbbb89c08e99181595eea296a2d291a2ea1f9e9f"} Oct 29 22:30:04.736196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:04.725297 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podStartSLOduration=1.725276344 podStartE2EDuration="1.725276344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:04.704656654 +0000 UTC m=+920.020050735" watchObservedRunningTime="2026-10-29 22:30:04.725276344 +0000 UTC m=+920.040670424" Oct 29 22:30:05.487441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:05.487412 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:05.488875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:05.488852 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 29 22:30:05.488962 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:05.488900 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 29 22:30:06.487752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.487722 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 29 22:30:06.488075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.487772 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 29 22:30:06.649845 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.649761 5073 generic.go:356] "Generic (PLEG): container finished" podID="9af72fb4-d271-4ab1-b369-ed035220d463" containerID="81e5e0901fd65ee108798504cb755de4aa9366e30bdcc3a9e762ca237db0202e" exitCode=60 Oct 29 22:30:06.649845 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.649823 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" event={"ID":"9af72fb4-d271-4ab1-b369-ed035220d463","Type":"ContainerDied","Data":"81e5e0901fd65ee108798504cb755de4aa9366e30bdcc3a9e762ca237db0202e"} Oct 29 22:30:06.650189 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.650170 5073 scope.go:122] "RemoveContainer" containerID="81e5e0901fd65ee108798504cb755de4aa9366e30bdcc3a9e762ca237db0202e" Oct 29 22:30:06.651553 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.651529 5073 generic.go:356] "Generic (PLEG): container finished" podID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerID="f8f9d5212755d4233421c2ecc87c29898952b5d1a6192327e152d958e512e52d" exitCode=60 Oct 29 22:30:06.651665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.651587 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" event={"ID":"681e0d8b-112f-4de4-aea4-1cff02bd87f6","Type":"ContainerDied","Data":"f8f9d5212755d4233421c2ecc87c29898952b5d1a6192327e152d958e512e52d"} Oct 29 22:30:06.651855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.651839 5073 scope.go:122] "RemoveContainer" containerID="f8f9d5212755d4233421c2ecc87c29898952b5d1a6192327e152d958e512e52d" Oct 29 22:30:06.653365 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.653336 5073 generic.go:356] "Generic (PLEG): container finished" podID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerID="49d80b02914d74ae140aea18666ce866f5b51ac3266437fd45ae5dd9321f84fe" exitCode=60 Oct 29 22:30:06.653469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.653368 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" event={"ID":"a6084d17-c3cb-405b-9c17-d628eec47d49","Type":"ContainerDied","Data":"49d80b02914d74ae140aea18666ce866f5b51ac3266437fd45ae5dd9321f84fe"} Oct 29 22:30:06.653661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.653638 5073 scope.go:122] "RemoveContainer" containerID="49d80b02914d74ae140aea18666ce866f5b51ac3266437fd45ae5dd9321f84fe" Oct 29 22:30:06.655257 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.655233 5073 generic.go:356] "Generic (PLEG): container finished" podID="3344d5cb-ce63-441b-a72f-a34dc68b0e01" containerID="74d736d8acb43a23a76a962ffd096d5b06c947f3bbe35427fe8cea6a5d1e48ed" exitCode=0 Oct 29 22:30:06.655361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:06.655305 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" event={"ID":"3344d5cb-ce63-441b-a72f-a34dc68b0e01","Type":"ContainerDied","Data":"74d736d8acb43a23a76a962ffd096d5b06c947f3bbe35427fe8cea6a5d1e48ed"} Oct 29 22:30:07.487432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.487397 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 29 22:30:07.487634 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.487453 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 29 22:30:07.662283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.662251 5073 generic.go:356] "Generic (PLEG): container finished" podID="9af72fb4-d271-4ab1-b369-ed035220d463" containerID="022006b8bec2005d9c89da4130221a8b2c4886dabb5ab24df886af9d7a7cdd4f" exitCode=60 Oct 29 22:30:07.662661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.662330 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" event={"ID":"9af72fb4-d271-4ab1-b369-ed035220d463","Type":"ContainerDied","Data":"022006b8bec2005d9c89da4130221a8b2c4886dabb5ab24df886af9d7a7cdd4f"} Oct 29 22:30:07.662661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.662431 5073 scope.go:122] "RemoveContainer" containerID="81e5e0901fd65ee108798504cb755de4aa9366e30bdcc3a9e762ca237db0202e" Oct 29 22:30:07.662772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.662717 5073 scope.go:122] "RemoveContainer" containerID="022006b8bec2005d9c89da4130221a8b2c4886dabb5ab24df886af9d7a7cdd4f" Oct 29 22:30:07.663042 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:07.663021 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-4kznj_odh-ai-gateway-infra(9af72fb4-d271-4ab1-b369-ed035220d463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" Oct 29 22:30:07.664406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.664384 5073 generic.go:356] "Generic (PLEG): container finished" podID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerID="46f68f2010c430990756a0f5cbd0bf51a101dcb568b445e98122a342804f4c37" exitCode=60 Oct 29 22:30:07.664503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.664426 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" event={"ID":"681e0d8b-112f-4de4-aea4-1cff02bd87f6","Type":"ContainerDied","Data":"46f68f2010c430990756a0f5cbd0bf51a101dcb568b445e98122a342804f4c37"} Oct 29 22:30:07.664694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.664677 5073 scope.go:122] "RemoveContainer" containerID="46f68f2010c430990756a0f5cbd0bf51a101dcb568b445e98122a342804f4c37" Oct 29 22:30:07.664958 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:07.664936 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4_odh-ai-gateway-infra(681e0d8b-112f-4de4-aea4-1cff02bd87f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" Oct 29 22:30:07.666456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.666430 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" event={"ID":"a6084d17-c3cb-405b-9c17-d628eec47d49","Type":"ContainerStarted","Data":"bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0"} Oct 29 22:30:07.682778 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.682756 5073 scope.go:122] "RemoveContainer" containerID="f8f9d5212755d4233421c2ecc87c29898952b5d1a6192327e152d958e512e52d" Oct 29 22:30:07.758738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.757802 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" podStartSLOduration=4.053861999 podStartE2EDuration="7.757786511s" totalImagesPullingTime="3.703924512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:07.755007406 +0000 UTC m=+923.070401487" watchObservedRunningTime="2026-10-29 22:30:07.757786511 +0000 UTC m=+923.073180607" Oct 29 22:30:07.869948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.869921 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:07.893749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.893718 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj\" (UniqueName: \"kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj\") pod \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " Oct 29 22:30:07.893896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.893810 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca\") pod \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " Oct 29 22:30:07.893896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.893862 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca\") pod \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\" (UID: \"3344d5cb-ce63-441b-a72f-a34dc68b0e01\") " Oct 29 22:30:07.894219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.894196 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca" pod "3344d5cb-ce63-441b-a72f-a34dc68b0e01" (UID: "3344d5cb-ce63-441b-a72f-a34dc68b0e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:07.894288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.894269 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:07.896734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.896714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj" pod "3344d5cb-ce63-441b-a72f-a34dc68b0e01" (UID: "3344d5cb-ce63-441b-a72f-a34dc68b0e01"). InnerVolumeSpecName "kube-api-access-chrnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:07.896734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.896712 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca" pod "3344d5cb-ce63-441b-a72f-a34dc68b0e01" (UID: "3344d5cb-ce63-441b-a72f-a34dc68b0e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:07.994922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.994890 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrnj\" (UniqueName: \"kubernetes.io/projected/3344d5cb-ce63-441b-a72f-a34dc68b0e01-kube-api-access-chrnj\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:07.994922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:07.994922 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3344d5cb-ce63-441b-a72f-a34dc68b0e01-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:08.137192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.137097 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs"] Oct 29 22:30:08.137707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.137685 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3344d5cb-ce63-441b-a72f-a34dc68b0e01" containerName="revoke-keys" Oct 29 22:30:08.137707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.137705 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3344d5cb-ce63-441b-a72f-a34dc68b0e01" containerName="revoke-keys" Oct 29 22:30:08.137871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.137860 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3344d5cb-ce63-441b-a72f-a34dc68b0e01" containerName="revoke-keys" Oct 29 22:30:08.141295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.141277 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.159601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.159579 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs"] Oct 29 22:30:08.197320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.197293 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.197463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.197345 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwj8j\" (UniqueName: \"kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.197540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.197480 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.298137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.298106 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.298314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.298197 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwj8j\" (UniqueName: \"kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.298314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.298233 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.299011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.298986 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.300706 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.300683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.335348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.335322 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwj8j\" (UniqueName: \"kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j\") pod \"maas-api-revoke-sub-83725a30-cn8zs\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.460330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.460301 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:08.487836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.487796 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 29 22:30:08.487943 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.487874 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 29 22:30:08.673982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.673952 5073 scope.go:122] "RemoveContainer" containerID="022006b8bec2005d9c89da4130221a8b2c4886dabb5ab24df886af9d7a7cdd4f" Oct 29 22:30:08.674774 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:08.674748 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-4kznj_odh-ai-gateway-infra(9af72fb4-d271-4ab1-b369-ed035220d463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" Oct 29 22:30:08.676592 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.676510 5073 scope.go:122] "RemoveContainer" containerID="46f68f2010c430990756a0f5cbd0bf51a101dcb568b445e98122a342804f4c37" Oct 29 22:30:08.676982 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:08.676945 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4_odh-ai-gateway-infra(681e0d8b-112f-4de4-aea4-1cff02bd87f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" Oct 29 22:30:08.677775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.677754 5073 generic.go:356] "Generic (PLEG): container finished" podID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerID="bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0" exitCode=60 Oct 29 22:30:08.677862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.677826 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" event={"ID":"a6084d17-c3cb-405b-9c17-d628eec47d49","Type":"ContainerDied","Data":"bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0"} Oct 29 22:30:08.677905 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.677868 5073 scope.go:122] "RemoveContainer" containerID="49d80b02914d74ae140aea18666ce866f5b51ac3266437fd45ae5dd9321f84fe" Oct 29 22:30:08.678142 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.678129 5073 scope.go:122] "RemoveContainer" containerID="bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0" Oct 29 22:30:08.678422 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:08.678406 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj_odh-ai-gateway-infra(a6084d17-c3cb-405b-9c17-d628eec47d49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" Oct 29 22:30:08.680586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.680563 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" event={"ID":"3344d5cb-ce63-441b-a72f-a34dc68b0e01","Type":"ContainerDied","Data":"c057dd3507cbcdd82d5c1b546280af236cb698e88aa0e05e2d05283dc72d1a0f"} Oct 29 22:30:08.680682 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.680648 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c057dd3507cbcdd82d5c1b546280af236cb698e88aa0e05e2d05283dc72d1a0f" Oct 29 22:30:08.680769 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.680755 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h" Oct 29 22:30:08.776914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:08.776883 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs"] Oct 29 22:30:09.258244 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:09.258212 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:30:09.488000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:09.487969 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 29 22:30:09.488182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:09.488024 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 29 22:30:09.691683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:09.691647 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" event={"ID":"c12bd78b-7c5f-4b60-987e-97688149a7e9","Type":"ContainerStarted","Data":"435aca19a9ab8e93b359afeec6717cb1c534ac09e63a67b198ec98848380ce95"} Oct 29 22:30:09.692120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:09.691700 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" event={"ID":"c12bd78b-7c5f-4b60-987e-97688149a7e9","Type":"ContainerStarted","Data":"d6862ed622fcd4f0d100e3c12170bcfbfdf02e26b531573dd77814f6d1ca8c3f"} Oct 29 22:30:09.693755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:09.693734 5073 scope.go:122] "RemoveContainer" containerID="bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0" Oct 29 22:30:09.693965 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:09.693947 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj_odh-ai-gateway-infra(a6084d17-c3cb-405b-9c17-d628eec47d49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" Oct 29 22:30:09.734983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:09.734931 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" podStartSLOduration=1.73490573 podStartE2EDuration="1.73490573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:09.734512548 +0000 UTC m=+925.049906629" watchObservedRunningTime="2026-10-29 22:30:09.73490573 +0000 UTC m=+925.050300002" Oct 29 22:30:10.680772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:10.680737 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:10.681112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:10.681081 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:10.682575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:10.682551 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 22:30:10.700562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:10.700539 5073 generic.go:356] "Generic (PLEG): container finished" podID="c12bd78b-7c5f-4b60-987e-97688149a7e9" containerID="435aca19a9ab8e93b359afeec6717cb1c534ac09e63a67b198ec98848380ce95" exitCode=0 Oct 29 22:30:10.700899 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:10.700568 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" event={"ID":"c12bd78b-7c5f-4b60-987e-97688149a7e9","Type":"ContainerDied","Data":"435aca19a9ab8e93b359afeec6717cb1c534ac09e63a67b198ec98848380ce95"} Oct 29 22:30:11.866674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.866650 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:11.960915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.960837 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca\") pod \"c12bd78b-7c5f-4b60-987e-97688149a7e9\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " Oct 29 22:30:11.961075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.960956 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j\" (UniqueName: \"kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j\") pod \"c12bd78b-7c5f-4b60-987e-97688149a7e9\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " Oct 29 22:30:11.961075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.960998 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca\") pod \"c12bd78b-7c5f-4b60-987e-97688149a7e9\" (UID: \"c12bd78b-7c5f-4b60-987e-97688149a7e9\") " Oct 29 22:30:11.961276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.961251 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca" pod "c12bd78b-7c5f-4b60-987e-97688149a7e9" (UID: "c12bd78b-7c5f-4b60-987e-97688149a7e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:11.963132 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.963109 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j" pod "c12bd78b-7c5f-4b60-987e-97688149a7e9" (UID: "c12bd78b-7c5f-4b60-987e-97688149a7e9"). InnerVolumeSpecName "kube-api-access-zwj8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:11.963251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:11.963201 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca" pod "c12bd78b-7c5f-4b60-987e-97688149a7e9" (UID: "c12bd78b-7c5f-4b60-987e-97688149a7e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:12.062496 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.062468 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwj8j\" (UniqueName: \"kubernetes.io/projected/c12bd78b-7c5f-4b60-987e-97688149a7e9-kube-api-access-zwj8j\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:12.062496 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.062495 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:12.062691 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.062506 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bd78b-7c5f-4b60-987e-97688149a7e9-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:12.615557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.615525 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:30:12.713393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.713366 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" Oct 29 22:30:12.713563 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.713410 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs" event={"ID":"c12bd78b-7c5f-4b60-987e-97688149a7e9","Type":"ContainerDied","Data":"d6862ed622fcd4f0d100e3c12170bcfbfdf02e26b531573dd77814f6d1ca8c3f"} Oct 29 22:30:12.713563 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:12.713467 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6862ed622fcd4f0d100e3c12170bcfbfdf02e26b531573dd77814f6d1ca8c3f" Oct 29 22:30:18.832068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:18.832036 5073 scope.go:122] "RemoveContainer" containerID="bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0" Oct 29 22:30:18.832591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:18.832292 5073 scope.go:122] "RemoveContainer" containerID="022006b8bec2005d9c89da4130221a8b2c4886dabb5ab24df886af9d7a7cdd4f" Oct 29 22:30:19.770456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:19.770418 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" event={"ID":"9af72fb4-d271-4ab1-b369-ed035220d463","Type":"ContainerStarted","Data":"92bde7018668e2d2cdf7e809e9fc9d45efb1e3da6de9971ef6fadec734f9c80b"} Oct 29 22:30:19.772222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:19.772164 5073 generic.go:356] "Generic (PLEG): container finished" podID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerID="bba070e0efe2ab2a33ac944ea6fa3e8c5a349ce938b50a22b07d229c1dca2640" exitCode=60 Oct 29 22:30:19.772222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:19.772181 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" event={"ID":"a6084d17-c3cb-405b-9c17-d628eec47d49","Type":"ContainerDied","Data":"bba070e0efe2ab2a33ac944ea6fa3e8c5a349ce938b50a22b07d229c1dca2640"} Oct 29 22:30:19.772402 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:19.772235 5073 scope.go:122] "RemoveContainer" containerID="bd34881a63448ecc0a2fb28bb2416343672187a2ff395d66a38c616ae952f5f0" Oct 29 22:30:19.772542 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:19.772527 5073 scope.go:122] "RemoveContainer" containerID="bba070e0efe2ab2a33ac944ea6fa3e8c5a349ce938b50a22b07d229c1dca2640" Oct 29 22:30:19.772808 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:19.772781 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj_odh-ai-gateway-infra(a6084d17-c3cb-405b-9c17-d628eec47d49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" Oct 29 22:30:19.814440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:19.814385 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" podStartSLOduration=16.210204553 podStartE2EDuration="19.814367728s" totalImagesPullingTime="3.604163175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:19.813173701 +0000 UTC m=+935.128567782" watchObservedRunningTime="2026-10-29 22:30:19.814367728 +0000 UTC m=+935.129761809" Oct 29 22:30:20.780498 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:20.780466 5073 generic.go:356] "Generic (PLEG): container finished" podID="9af72fb4-d271-4ab1-b369-ed035220d463" containerID="92bde7018668e2d2cdf7e809e9fc9d45efb1e3da6de9971ef6fadec734f9c80b" exitCode=60 Oct 29 22:30:20.780979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:20.780550 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" event={"ID":"9af72fb4-d271-4ab1-b369-ed035220d463","Type":"ContainerDied","Data":"92bde7018668e2d2cdf7e809e9fc9d45efb1e3da6de9971ef6fadec734f9c80b"} Oct 29 22:30:20.780979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:20.780634 5073 scope.go:122] "RemoveContainer" containerID="022006b8bec2005d9c89da4130221a8b2c4886dabb5ab24df886af9d7a7cdd4f" Oct 29 22:30:20.780979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:20.780973 5073 scope.go:122] "RemoveContainer" containerID="92bde7018668e2d2cdf7e809e9fc9d45efb1e3da6de9971ef6fadec734f9c80b" Oct 29 22:30:20.781218 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:20.781194 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-4kznj_odh-ai-gateway-infra(9af72fb4-d271-4ab1-b369-ed035220d463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" Oct 29 22:30:20.826823 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:20.826796 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj"] Oct 29 22:30:20.900993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:20.900965 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj"] Oct 29 22:30:21.066748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.066723 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:21.179209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.179177 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln\" (UniqueName: \"kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln\") pod \"a6084d17-c3cb-405b-9c17-d628eec47d49\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " Oct 29 22:30:21.179357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.179266 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca\") pod \"a6084d17-c3cb-405b-9c17-d628eec47d49\" (UID: \"a6084d17-c3cb-405b-9c17-d628eec47d49\") " Oct 29 22:30:21.179600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.179568 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca" pod "a6084d17-c3cb-405b-9c17-d628eec47d49" (UID: "a6084d17-c3cb-405b-9c17-d628eec47d49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:21.179753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.179738 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6084d17-c3cb-405b-9c17-d628eec47d49-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.181309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.181288 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln" pod "a6084d17-c3cb-405b-9c17-d628eec47d49" (UID: "a6084d17-c3cb-405b-9c17-d628eec47d49"). InnerVolumeSpecName "kube-api-access-6zxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:21.280340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.280309 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxln\" (UniqueName: \"kubernetes.io/projected/a6084d17-c3cb-405b-9c17-d628eec47d49-kube-api-access-6zxln\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.791443 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.791417 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" Oct 29 22:30:21.791852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.791448 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj" event={"ID":"a6084d17-c3cb-405b-9c17-d628eec47d49","Type":"ContainerDied","Data":"d1339fdf1749f3038cc1b4a2592a688f6341eef9d98b594beab11a1d833da442"} Oct 29 22:30:21.791852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.791556 5073 scope.go:122] "RemoveContainer" containerID="bba070e0efe2ab2a33ac944ea6fa3e8c5a349ce938b50a22b07d229c1dca2640" Oct 29 22:30:21.831127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.831096 5073 scope.go:122] "RemoveContainer" containerID="46f68f2010c430990756a0f5cbd0bf51a101dcb568b445e98122a342804f4c37" Oct 29 22:30:21.874183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.872492 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj"] Oct 29 22:30:21.877604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.877578 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d3632-29888550-vtcwj"] Oct 29 22:30:21.949940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:21.949917 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:22.088571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.088508 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb\" (UniqueName: \"kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb\") pod \"9af72fb4-d271-4ab1-b369-ed035220d463\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " Oct 29 22:30:22.088730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.088675 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca\") pod \"9af72fb4-d271-4ab1-b369-ed035220d463\" (UID: \"9af72fb4-d271-4ab1-b369-ed035220d463\") " Oct 29 22:30:22.089010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.088986 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca" pod "9af72fb4-d271-4ab1-b369-ed035220d463" (UID: "9af72fb4-d271-4ab1-b369-ed035220d463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:22.108381 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.108357 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb" pod "9af72fb4-d271-4ab1-b369-ed035220d463" (UID: "9af72fb4-d271-4ab1-b369-ed035220d463"). InnerVolumeSpecName "kube-api-access-29zqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:22.189979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.189946 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af72fb4-d271-4ab1-b369-ed035220d463-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.189979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.189982 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zqb\" (UniqueName: \"kubernetes.io/projected/9af72fb4-d271-4ab1-b369-ed035220d463-kube-api-access-29zqb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.801590 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.801562 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" Oct 29 22:30:22.802071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.801562 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj" event={"ID":"9af72fb4-d271-4ab1-b369-ed035220d463","Type":"ContainerDied","Data":"df826c655f2ed557510c60a43b396635b1a090962ecfdadbef30740bbf0350d5"} Oct 29 22:30:22.802183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.802116 5073 scope.go:122] "RemoveContainer" containerID="92bde7018668e2d2cdf7e809e9fc9d45efb1e3da6de9971ef6fadec734f9c80b" Oct 29 22:30:22.805459 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.805420 5073 generic.go:356] "Generic (PLEG): container finished" podID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerID="e89c531f0505902e55ecd025d52792b0aafda8534990a0dcc831a8dcfca147f3" exitCode=60 Oct 29 22:30:22.805600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.805463 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" event={"ID":"681e0d8b-112f-4de4-aea4-1cff02bd87f6","Type":"ContainerDied","Data":"e89c531f0505902e55ecd025d52792b0aafda8534990a0dcc831a8dcfca147f3"} Oct 29 22:30:22.805835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.805817 5073 scope.go:122] "RemoveContainer" containerID="e89c531f0505902e55ecd025d52792b0aafda8534990a0dcc831a8dcfca147f3" Oct 29 22:30:22.806370 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:22.806062 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4_odh-ai-gateway-infra(681e0d8b-112f-4de4-aea4-1cff02bd87f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" Oct 29 22:30:22.814269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.814248 5073 scope.go:122] "RemoveContainer" containerID="46f68f2010c430990756a0f5cbd0bf51a101dcb568b445e98122a342804f4c37" Oct 29 22:30:22.838209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.838142 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" path="/var/lib/kubelet/pods/a6084d17-c3cb-405b-9c17-d628eec47d49/volumes" Oct 29 22:30:22.903896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.903870 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj"] Oct 29 22:30:22.908815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:22.908788 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-4kznj"] Oct 29 22:30:23.871613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:23.871581 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4"] Oct 29 22:30:24.018494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.018471 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:24.109508 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.109473 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca\") pod \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " Oct 29 22:30:24.109759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.109738 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf\" (UniqueName: \"kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf\") pod \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\" (UID: \"681e0d8b-112f-4de4-aea4-1cff02bd87f6\") " Oct 29 22:30:24.109990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.109965 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca" pod "681e0d8b-112f-4de4-aea4-1cff02bd87f6" (UID: "681e0d8b-112f-4de4-aea4-1cff02bd87f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:24.110213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.110194 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e0d8b-112f-4de4-aea4-1cff02bd87f6-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:24.112000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.111980 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf" pod "681e0d8b-112f-4de4-aea4-1cff02bd87f6" (UID: "681e0d8b-112f-4de4-aea4-1cff02bd87f6"). InnerVolumeSpecName "kube-api-access-jvqxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:24.210817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.210742 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqxf\" (UniqueName: \"kubernetes.io/projected/681e0d8b-112f-4de4-aea4-1cff02bd87f6-kube-api-access-jvqxf\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:30:24.820908 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.820880 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" Oct 29 22:30:24.821078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.820924 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4" event={"ID":"681e0d8b-112f-4de4-aea4-1cff02bd87f6","Type":"ContainerDied","Data":"62523c111a028473416901370b00dd45f19c6ddc887770deb8019cb40742734b"} Oct 29 22:30:24.821078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.820977 5073 scope.go:122] "RemoveContainer" containerID="e89c531f0505902e55ecd025d52792b0aafda8534990a0dcc831a8dcfca147f3" Oct 29 22:30:24.847896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.847871 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" path="/var/lib/kubelet/pods/9af72fb4-d271-4ab1-b369-ed035220d463/volumes" Oct 29 22:30:24.886663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.886640 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4"] Oct 29 22:30:24.895854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:24.895831 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888550-b4bf4"] Oct 29 22:30:26.838262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:26.838218 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" path="/var/lib/kubelet/pods/681e0d8b-112f-4de4-aea4-1cff02bd87f6/volumes" Oct 29 22:30:32.977288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977257 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl"] Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977709 5073 cpu_manager.go:414] "containerMap: removing container" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977720 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977739 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977744 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977754 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977759 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977769 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977773 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977780 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977785 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977791 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977795 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977806 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977811 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977827 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c12bd78b-7c5f-4b60-987e-97688149a7e9" containerName="revoke-keys" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977832 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12bd78b-7c5f-4b60-987e-97688149a7e9" containerName="revoke-keys" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977840 5073 cpu_manager.go:414] "containerMap: removing container" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977844 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977927 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977936 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977941 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977950 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12bd78b-7c5f-4b60-987e-97688149a7e9" containerName="revoke-keys" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977956 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977961 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977968 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:32.979656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.977976 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:32.981072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:32.981056 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.011435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.011409 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8d3632-openshift-default-dockercfg-x5w92\"" Oct 29 22:30:33.014879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.014855 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl"] Oct 29 22:30:33.046814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.046787 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw"] Oct 29 22:30:33.047361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.047345 5073 cpu_manager.go:414] "containerMap: removing container" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:33.047421 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.047364 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e0d8b-112f-4de4-aea4-1cff02bd87f6" containerName="cleanup" Oct 29 22:30:33.047471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.047463 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6084d17-c3cb-405b-9c17-d628eec47d49" containerName="cleanup" Oct 29 22:30:33.047507 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.047472 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af72fb4-d271-4ab1-b369-ed035220d463" containerName="cleanup" Oct 29 22:30:33.050708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.050690 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.083821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.083790 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw"] Oct 29 22:30:33.097877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.097839 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4a0287-openshift-default-dockercfg-dcw9p\"" Oct 29 22:30:33.100908 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.100881 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvzv\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.100939 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.100974 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.101004 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101160 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.101053 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101160 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.101100 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101160 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.101130 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.101192 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.101274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.101221 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202241 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202275 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202295 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202320 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202368 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202391 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202431 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202453 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202567 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202603 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvzv\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202643 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24qp\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202745 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.202919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202862 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.202919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202907 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.203127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202942 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.203127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202977 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.203127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.202997 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.203127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.203065 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.203364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.203127 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.203364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.203220 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.203364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.203316 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.203364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.203344 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.204970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.204940 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.205534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.205514 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.222836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.222808 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvzv\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.223384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.223362 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.294866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.294815 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:33.304031 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304012 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304104 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304061 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304104 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304097 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304182 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304245 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304230 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24qp\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304264 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304307 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304360 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304402 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304457 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304476 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304690 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.304780 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.304711 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.305434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.305406 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.306631 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.306611 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.306711 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.306656 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.324637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.324608 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.324918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.324903 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24qp\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.365184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.364444 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:33.368360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.367744 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs"] Oct 29 22:30:33.372628 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.372591 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.390580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.390525 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-92fd86-openshift-default-dockercfg-485tp\"" Oct 29 22:30:33.392385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.392363 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs"] Oct 29 22:30:33.506940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.506900 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.506969 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507065 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507133 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507226 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507337 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507403 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqns\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507459 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.507514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.507497 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.605672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.605647 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl"] Oct 29 22:30:33.606302 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:33.606280 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5223d9bb_d03a_473c_9d24_876c69f2dbe3.slice/crio-500e356f4cab9227fc52e9defb120e631235bc0acb5eaddbd58631aa9498cb8d WatchSource:0}: Error finding container 500e356f4cab9227fc52e9defb120e631235bc0acb5eaddbd58631aa9498cb8d: Status 404 returned error can't find the container with id 500e356f4cab9227fc52e9defb120e631235bc0acb5eaddbd58631aa9498cb8d Oct 29 22:30:33.608039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608021 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608049 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608141 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608083 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608221 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608204 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608239 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608327 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608298 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608333 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608459 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608414 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608475 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqns\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608493 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608736 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:33.608906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608849 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.608906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.608895 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.609083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.609052 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.609192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.609098 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:33.609192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.609175 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:33.610307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.610286 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.611258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.611238 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.611464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.611381 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.630032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.630003 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.630302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.630286 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqns\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.691076 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:33.691033 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7fbd69_c165_41e7_a768_2443557073ff.slice/crio-012059c1b144d3393ca03d7d93441661b895b3db85d60d54e58c827599438b11 WatchSource:0}: Error finding container 012059c1b144d3393ca03d7d93441661b895b3db85d60d54e58c827599438b11: Status 404 returned error can't find the container with id 012059c1b144d3393ca03d7d93441661b895b3db85d60d54e58c827599438b11 Oct 29 22:30:33.691213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.691128 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw"] Oct 29 22:30:33.693446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.693416 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:33.693557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.693474 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:33.693557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.693502 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:33.697705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.697511 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:33.884917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.884830 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" event={"ID":"5223d9bb-d03a-473c-9d24-876c69f2dbe3","Type":"ContainerStarted","Data":"e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651"} Oct 29 22:30:33.884917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.884905 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" event={"ID":"5223d9bb-d03a-473c-9d24-876c69f2dbe3","Type":"ContainerStarted","Data":"500e356f4cab9227fc52e9defb120e631235bc0acb5eaddbd58631aa9498cb8d"} Oct 29 22:30:33.887626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.887593 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" event={"ID":"0b7fbd69-c165-41e7-a768-2443557073ff","Type":"ContainerStarted","Data":"137ff07fcf9251bd0648c3f440399af83a57bceb9612cce43a498fe6209ebe45"} Oct 29 22:30:33.887753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.887650 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" event={"ID":"0b7fbd69-c165-41e7-a768-2443557073ff","Type":"ContainerStarted","Data":"012059c1b144d3393ca03d7d93441661b895b3db85d60d54e58c827599438b11"} Oct 29 22:30:33.931300 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.931249 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podStartSLOduration=1.931236183 podStartE2EDuration="1.931236183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:33.929604888 +0000 UTC m=+949.244998995" watchObservedRunningTime="2026-10-29 22:30:33.931236183 +0000 UTC m=+949.246630262" Oct 29 22:30:33.976045 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:33.975992 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podStartSLOduration=0.975979259 podStartE2EDuration="975.979259ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:33.973324924 +0000 UTC m=+949.288719039" watchObservedRunningTime="2026-10-29 22:30:33.975979259 +0000 UTC m=+949.291373400" Oct 29 22:30:34.022468 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.022104 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs"] Oct 29 22:30:34.024221 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:30:34.024195 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37686afe_62ab_4be1_80d0_ec40cad5ece7.slice/crio-da2c99b60a0fd5426e95fb38d6b170ef5f30c6e05b037aa9f7c79f0da438ddcb WatchSource:0}: Error finding container da2c99b60a0fd5426e95fb38d6b170ef5f30c6e05b037aa9f7c79f0da438ddcb: Status 404 returned error can't find the container with id da2c99b60a0fd5426e95fb38d6b170ef5f30c6e05b037aa9f7c79f0da438ddcb Oct 29 22:30:34.026717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.026685 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:34.026820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.026748 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:34.026820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.026779 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:34.295455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.295420 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:34.296982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.296962 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:34.297035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.297006 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:34.367358 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.367326 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:34.368794 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.368762 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:34.368914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.368821 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:34.894542 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.894508 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" event={"ID":"37686afe-62ab-4be1-80d0-ec40cad5ece7","Type":"ContainerStarted","Data":"ca401e12bd329f5a027088c850e0e32cefc8361500759de4e37ea3e1a6e2d5a3"} Oct 29 22:30:34.894721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.894562 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" event={"ID":"37686afe-62ab-4be1-80d0-ec40cad5ece7","Type":"ContainerStarted","Data":"da2c99b60a0fd5426e95fb38d6b170ef5f30c6e05b037aa9f7c79f0da438ddcb"} Oct 29 22:30:34.974298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:34.974238 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podStartSLOduration=1.974218206 podStartE2EDuration="1.974218206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:34.971090178 +0000 UTC m=+950.286484259" watchObservedRunningTime="2026-10-29 22:30:34.974218206 +0000 UTC m=+950.289612289" Oct 29 22:30:35.295525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.295488 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:35.295992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.295965 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:35.369177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.368594 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:35.369177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.368665 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:35.698458 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.698373 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:35.699858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.699836 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:30:35.699913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:35.699878 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:30:36.295707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:36.295666 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:36.296252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:36.295722 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:36.368206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:36.367562 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:36.368206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:36.367624 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:36.698900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:36.698857 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:30:36.699044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:36.698925 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:30:37.295987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.295948 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:37.296414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.296028 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:37.373647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.373610 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:37.373907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.373877 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:37.698669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.698578 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:30:37.698669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.698640 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:30:37.802535 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:37.802493 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:30:38.297177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:38.297125 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:38.297778 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:38.297747 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:38.369187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:38.367663 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:38.369187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:38.367725 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:38.698384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:38.698285 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:30:38.698384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:38.698365 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:30:39.300609 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.300573 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:39.301185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.301135 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:39.366354 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:30:39.366317 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:30:39.367339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.367256 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:39.367521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.367503 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:39.704251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.701526 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:30:39.704251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.701634 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:30:39.972275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:39.972227 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:30:40.053324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.053274 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:30:40.106504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.106461 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:30:40.295498 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.295409 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:30:40.295756 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.295725 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 29 22:30:40.368751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.368708 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:30:40.369404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.369372 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:30:40.698304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.698213 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:30:40.698304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:40.698276 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:30:41.431068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:41.431022 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:41.432110 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:41.432084 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:41.433338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:41.433315 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:30:42.237556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.237521 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:42.237955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.237920 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:42.300317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.300286 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:30:42.739306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.739277 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:30:42.739781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.739331 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:30:42.791016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.790986 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:42.791500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.791473 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:42.792351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:42.792329 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:30:55.520127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:30:55.520086 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:31:03.337216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.337184 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s"] Oct 29 22:31:03.341164 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.341082 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.375181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.375140 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"distinct-w5-0d12c5-kserve-self-signed-certs\"" Oct 29 22:31:03.379655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.379628 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s"] Oct 29 22:31:03.469219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.469191 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.469377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.469225 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.469377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.469244 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.469377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.469350 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmbv\" (UniqueName: \"kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.469548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.469447 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.469548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.469518 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.570657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.570623 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.570657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.570663 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.570941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.570683 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.570941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.570728 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmbv\" (UniqueName: \"kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.570941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.570755 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.570941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.570787 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.571192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.571118 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.571504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.571481 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.571634 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.571505 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.574197 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.574116 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.574498 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.574452 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.600409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.599637 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmbv\" (UniqueName: \"kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv\") pod \"distinct-w5-0d12c5-kserve-86b7c56cff-b242s\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.654039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.654003 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:03.961278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:03.961253 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s"] Oct 29 22:31:03.964768 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:03.964741 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c2e6ae_8ac3_4286_8dad_d44e63acdb4d.slice/crio-ac629943d79ea2b873cc6e47043ed9e0c53c0ea467f644632fcea05235de47bb WatchSource:0}: Error finding container ac629943d79ea2b873cc6e47043ed9e0c53c0ea467f644632fcea05235de47bb: Status 404 returned error can't find the container with id ac629943d79ea2b873cc6e47043ed9e0c53c0ea467f644632fcea05235de47bb Oct 29 22:31:04.188397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:04.188362 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" event={"ID":"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d","Type":"ContainerStarted","Data":"f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5"} Oct 29 22:31:04.188397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:04.188401 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" event={"ID":"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d","Type":"ContainerStarted","Data":"ac629943d79ea2b873cc6e47043ed9e0c53c0ea467f644632fcea05235de47bb"} Oct 29 22:31:04.230591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:04.230502 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podStartSLOduration=1.230489411 podStartE2EDuration="1.230489411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:04.228824755 +0000 UTC m=+979.544218858" watchObservedRunningTime="2026-10-29 22:31:04.230489411 +0000 UTC m=+979.545883491" Oct 29 22:31:04.654934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:04.654851 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:09.254501 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:31:09.254460 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:31:12.888419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:12.888325 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:31:12.888419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:12.888393 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:13.654263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:13.654226 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:13.667091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:13.667065 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:14.266853 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:14.266825 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:31:17.509279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.509240 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g"] Oct 29 22:31:17.514650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.514625 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.531698 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.531673 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g"] Oct 29 22:31:17.542019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.541907 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"distinct-w4-4a0287-kserve-self-signed-certs\"" Oct 29 22:31:17.554525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.554502 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.554626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.554559 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.554675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.554624 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.554675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.554647 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.554754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.554740 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm768\" (UniqueName: \"kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.554796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.554786 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.655748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.655707 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.655937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.655802 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.655937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.655836 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.655937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.655860 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.656115 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.655965 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm768\" (UniqueName: \"kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.656115 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.656029 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.656252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.656165 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.656252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.656230 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.656363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.656309 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.658447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.658423 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.658600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.658474 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.676976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.676952 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm768\" (UniqueName: \"kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768\") pod \"distinct-w4-4a0287-kserve-6db95d94bf-v7r9g\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:17.830254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:17.830143 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:18.145452 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.145424 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g"] Oct 29 22:31:18.146906 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:18.146570 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30771aa7_e03f_4557_94e0_7b920bdc68ee.slice/crio-d3de7e926174029803856c7f25d86e30d28f02ce3969d1d390f64d760dbf21d2 WatchSource:0}: Error finding container d3de7e926174029803856c7f25d86e30d28f02ce3969d1d390f64d760dbf21d2: Status 404 returned error can't find the container with id d3de7e926174029803856c7f25d86e30d28f02ce3969d1d390f64d760dbf21d2 Oct 29 22:31:18.279318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.279281 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" event={"ID":"30771aa7-e03f-4557-94e0-7b920bdc68ee","Type":"ContainerStarted","Data":"791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4"} Oct 29 22:31:18.279474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.279330 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" event={"ID":"30771aa7-e03f-4557-94e0-7b920bdc68ee","Type":"ContainerStarted","Data":"d3de7e926174029803856c7f25d86e30d28f02ce3969d1d390f64d760dbf21d2"} Oct 29 22:31:18.321046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.321002 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podStartSLOduration=1.320987152 podStartE2EDuration="1.320987152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:18.320198684 +0000 UTC m=+993.635592765" watchObservedRunningTime="2026-10-29 22:31:18.320987152 +0000 UTC m=+993.636381231" Oct 29 22:31:18.674570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.674537 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45"] Oct 29 22:31:18.679185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.679126 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.701407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.701379 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45"] Oct 29 22:31:18.768662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.768626 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.768662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.768659 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.768871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.768689 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shw7m\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.768871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.768842 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.768980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.768870 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.768980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.768888 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.769079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.769039 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.769079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.769072 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.769197 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.769105 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.830649 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.830623 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:18.870332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870307 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870337 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870366 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870473 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870501 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870549 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shw7m\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870619 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870650 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870674 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870734 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.870865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.870790 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.871209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.871185 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.871328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.871219 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.871484 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.871384 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.873064 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.873044 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.873329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.873312 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.890627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.890604 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.890759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.890740 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shw7m\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:18.961241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.961214 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt"] Oct 29 22:31:18.966181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.966164 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:18.989769 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.989747 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt"] Oct 29 22:31:18.996261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:18.996238 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:19.072811 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.072783 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.072917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.072895 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6m2\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.072982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.072927 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.073036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.072980 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.073036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.073006 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.073137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.073070 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.073219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.073136 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.073275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.073225 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.073275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.073251 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174163 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174114 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6m2\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174182 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174290 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174438 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174320 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174438 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174401 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174471 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174543 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174561 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174701 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174634 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174808 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.174988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.174893 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.175061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.175035 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.175122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.175088 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.175265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.175248 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.176788 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.176768 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.177123 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.177102 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.194991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.194969 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.195285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.195264 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6m2\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.284720 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.284651 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:19.329016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.328983 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45"] Oct 29 22:31:19.330373 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:19.330349 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd77fe7_3293_4b49_9ca6_53e17e5f56be.slice/crio-9e6d0d16a0099842be2303ffabde62352823ccf206b9f8926c9f77d6e4b0851c WatchSource:0}: Error finding container 9e6d0d16a0099842be2303ffabde62352823ccf206b9f8926c9f77d6e4b0851c: Status 404 returned error can't find the container with id 9e6d0d16a0099842be2303ffabde62352823ccf206b9f8926c9f77d6e4b0851c Oct 29 22:31:19.333463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.333426 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:19.333568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.333505 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:19.333568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.333549 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:19.605572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.605545 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt"] Oct 29 22:31:19.607416 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:19.607387 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d986f4_1e69_416d_89a0_79f70a431e83.slice/crio-c8da38ea7e810a95ffe73ac8d74cf079125f5ee3810c94ce96ed6d165e366000 WatchSource:0}: Error finding container c8da38ea7e810a95ffe73ac8d74cf079125f5ee3810c94ce96ed6d165e366000: Status 404 returned error can't find the container with id c8da38ea7e810a95ffe73ac8d74cf079125f5ee3810c94ce96ed6d165e366000 Oct 29 22:31:19.610057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.610015 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:19.610163 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.610102 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:19.610208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:19.610189 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:20.293335 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.293297 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" event={"ID":"7dd77fe7-3293-4b49-9ca6-53e17e5f56be","Type":"ContainerStarted","Data":"1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3"} Oct 29 22:31:20.293810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.293355 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" event={"ID":"7dd77fe7-3293-4b49-9ca6-53e17e5f56be","Type":"ContainerStarted","Data":"9e6d0d16a0099842be2303ffabde62352823ccf206b9f8926c9f77d6e4b0851c"} Oct 29 22:31:20.295522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.295493 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" event={"ID":"60d986f4-1e69-416d-89a0-79f70a431e83","Type":"ContainerStarted","Data":"f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d"} Oct 29 22:31:20.295641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.295535 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" event={"ID":"60d986f4-1e69-416d-89a0-79f70a431e83","Type":"ContainerStarted","Data":"c8da38ea7e810a95ffe73ac8d74cf079125f5ee3810c94ce96ed6d165e366000"} Oct 29 22:31:20.355176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.354692 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podStartSLOduration=2.354670847 podStartE2EDuration="2.354670847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:20.341435416 +0000 UTC m=+995.656829519" watchObservedRunningTime="2026-10-29 22:31:20.354670847 +0000 UTC m=+995.670064932" Oct 29 22:31:20.394305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.394250 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podStartSLOduration=2.394235982 podStartE2EDuration="2.394235982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:20.390512433 +0000 UTC m=+995.705906523" watchObservedRunningTime="2026-10-29 22:31:20.394235982 +0000 UTC m=+995.709630092" Oct 29 22:31:20.997853 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.997820 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:20.999404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.999378 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:20.999531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:20.999433 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 29 22:31:21.285716 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:21.285639 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:21.287095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:21.287071 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.287218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:21.287111 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 29 22:31:21.997235 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:21.997196 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.997731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:21.997266 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 29 22:31:22.286197 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:22.286099 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.286342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:22.286236 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 29 22:31:22.997225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:22.997194 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.997392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:22.997246 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 29 22:31:23.285130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:23.285063 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.285130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:23.285107 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 29 22:31:23.997093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:23.997064 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.997308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:23.997245 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 29 22:31:24.285817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:24.285731 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.285817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:24.285792 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 29 22:31:24.997084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:24.997052 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.997311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:24.997105 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 29 22:31:25.285690 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:25.285611 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.285690 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:25.285665 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 29 22:31:25.997119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:25.997085 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.997345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:25.997133 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 29 22:31:26.669492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:26.669459 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:26.671130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:26.671097 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:26.672470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:26.672452 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:31:27.001459 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:27.001434 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:27.001911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:27.001892 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:27.002501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:27.002481 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:31:27.831269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:27.831234 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:27.844318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:27.844297 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:28.394257 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:28.394227 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:31:29.661616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.661585 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf"] Oct 29 22:31:29.665487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.665463 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.687426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.687397 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf"] Oct 29 22:31:29.693488 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.693466 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40cf9a\"/\"distinct-w0-40cf9a-kserve-self-signed-certs\"" Oct 29 22:31:29.749253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.749230 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.749415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.749271 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.749415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.749287 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.749415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.749306 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.749415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.749365 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kkqp\" (UniqueName: \"kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.749555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.749445 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.812990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.812954 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:31:29.851265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851226 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.851461 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851300 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.852251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851735 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.852251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851757 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.852251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851773 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.852251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851822 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kkqp\" (UniqueName: \"kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.852251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.851862 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.857175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.853573 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.857175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.854133 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.873834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.873804 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kkqp\" (UniqueName: \"kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.875752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.875722 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.876326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.876304 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs\") pod \"distinct-w0-40cf9a-kserve-dd7858b54-vvnmf\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:29.981127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:29.980880 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:30.323449 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:30.323426 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf"] Oct 29 22:31:30.324886 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:30.324858 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ce1d8ec_3c00_405f_a6d0_d5e520cc2d4c.slice/crio-48812a30a11338acecb87e7c605e5836581247f84325fb02cfb9a03a6a8c061b WatchSource:0}: Error finding container 48812a30a11338acecb87e7c605e5836581247f84325fb02cfb9a03a6a8c061b: Status 404 returned error can't find the container with id 48812a30a11338acecb87e7c605e5836581247f84325fb02cfb9a03a6a8c061b Oct 29 22:31:30.397627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:30.397595 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" event={"ID":"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c","Type":"ContainerStarted","Data":"48812a30a11338acecb87e7c605e5836581247f84325fb02cfb9a03a6a8c061b"} Oct 29 22:31:31.415748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:31.415707 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" event={"ID":"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c","Type":"ContainerStarted","Data":"784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5"} Oct 29 22:31:31.994178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:31.984464 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:33.411782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.411728 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podStartSLOduration=4.411713539 podStartE2EDuration="4.411713539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:31.468792579 +0000 UTC m=+1006.784186671" watchObservedRunningTime="2026-10-29 22:31:33.411713539 +0000 UTC m=+1008.727107619" Oct 29 22:31:33.412189 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.412075 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9"] Oct 29 22:31:33.416124 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.416108 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.447029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.446987 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9"] Oct 29 22:31:33.553752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.553724 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.553911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.553776 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.553953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.553935 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.553998 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.553953 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.554056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.553996 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.554056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.554012 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.554216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.554140 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.554289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.554230 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.554289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.554251 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs9l\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.654879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.654847 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.654883 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.654914 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.654945 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.654974 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs9l\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655051 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655119 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655282 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655291 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655310 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655336 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655684 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655790 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655763 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.655843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.655832 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.657802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.657780 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.658240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.658222 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.694135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.694075 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.694684 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.694668 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs9l\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.732301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.732277 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:33.866791 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.866760 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq"] Oct 29 22:31:33.871862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.871843 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.893078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.893051 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq"] Oct 29 22:31:33.960351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960327 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960385 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960401 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960479 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960544 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960620 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960673 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960645 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960673 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960665 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:33.960802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:33.960783 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrfb\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.047943 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:34.047912 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111ead0c_60ad_4fc1_8ac4_2ef6e56060db.slice/crio-8e4bb23bc1a26ff8350eca88f866f9fb96a8ec35fdba46dcd0f48f29b518cc44 WatchSource:0}: Error finding container 8e4bb23bc1a26ff8350eca88f866f9fb96a8ec35fdba46dcd0f48f29b518cc44: Status 404 returned error can't find the container with id 8e4bb23bc1a26ff8350eca88f866f9fb96a8ec35fdba46dcd0f48f29b518cc44 Oct 29 22:31:34.048758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.048731 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9"] Oct 29 22:31:34.050774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.050737 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:34.050862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.050834 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:34.050903 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.050873 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:34.061674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061649 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.061761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061684 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.061761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061713 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.061954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061806 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.061954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061849 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.061954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061888 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.061954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061924 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.061993 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrfb\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.062106 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.062178 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.062259 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.062279 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.062495 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.062676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.062657 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.064420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.064396 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.064568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.064545 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.082707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.082683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrfb\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.083100 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.083083 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.190637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.190607 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:34.444331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.444296 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" event={"ID":"111ead0c-60ad-4fc1-8ac4-2ef6e56060db","Type":"ContainerStarted","Data":"f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab"} Oct 29 22:31:34.444815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.444354 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" event={"ID":"111ead0c-60ad-4fc1-8ac4-2ef6e56060db","Type":"ContainerStarted","Data":"8e4bb23bc1a26ff8350eca88f866f9fb96a8ec35fdba46dcd0f48f29b518cc44"} Oct 29 22:31:34.490036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.489986 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podStartSLOduration=1.4899700980000001 podStartE2EDuration="1.489970098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:34.488460258 +0000 UTC m=+1009.803854340" watchObservedRunningTime="2026-10-29 22:31:34.489970098 +0000 UTC m=+1009.805364255" Oct 29 22:31:34.512630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.507879 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq"] Oct 29 22:31:34.521450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.520484 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:34.521450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.520577 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:34.521450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.520621 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:34.732469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:34.732424 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:35.470138 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:35.470091 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" event={"ID":"7559b915-afb7-4ceb-974b-8122f6f77760","Type":"ContainerStarted","Data":"47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8"} Oct 29 22:31:35.470613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:35.470190 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" event={"ID":"7559b915-afb7-4ceb-974b-8122f6f77760","Type":"ContainerStarted","Data":"6610e1f0f5f3b145507947eeab164e560de8e4cd2b334cf6cdcc38a097f682a2"} Oct 29 22:31:35.566261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:35.562589 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podStartSLOduration=2.5625667 podStartE2EDuration="2.5625667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:35.531995486 +0000 UTC m=+1010.847389581" watchObservedRunningTime="2026-10-29 22:31:35.5625667 +0000 UTC m=+1010.877960782" Oct 29 22:31:35.746448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:35.746331 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:31:35.746709 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:35.746664 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 22:31:36.191707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:36.191638 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:36.192682 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:36.192647 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:31:36.192736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:36.192708 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:31:36.736334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:36.736299 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:31:36.736902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:36.736872 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 22:31:37.191613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:37.191584 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:31:37.191796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:37.191630 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:31:37.748175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:37.747502 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:31:37.748175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:37.747560 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 22:31:38.190987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:38.190905 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:31:38.190987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:38.190953 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:31:38.736176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:38.736081 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:31:38.736368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:38.736179 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 22:31:39.192013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:39.191929 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:31:39.192013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:39.191990 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:31:39.285834 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:31:39.285801 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:31:39.733519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:39.733484 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:31:39.733788 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:39.733761 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 22:31:39.981892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:39.981854 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:39.995610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:39.995511 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:40.191615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:40.191577 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:31:40.191833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:40.191639 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:31:40.561596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:40.561565 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:31:41.075671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:41.075640 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:41.076724 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:41.076698 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:41.077592 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:41.077567 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:31:41.192570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:41.192538 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:31:41.192841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:41.192814 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:31:42.195740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:42.195709 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:42.196250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:42.196181 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:42.196965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:42.196947 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:31:48.266803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.266773 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr"] Oct 29 22:31:48.272401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.272379 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.291886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.291863 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr"] Oct 29 22:31:48.364090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364056 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364089 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzz4j\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364210 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364265 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364329 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364405 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364469 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364516 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.364579 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.364568 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465329 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465380 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465396 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzz4j\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465456 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465504 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465519 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465557 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465611 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465631 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.465821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465713 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.466044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.465982 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.466105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.466051 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.466324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.466301 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.466560 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.466544 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.468250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.468230 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.468322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.468263 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.487102 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.487078 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzz4j\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.487332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.487312 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.588275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.588183 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:48.906571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.906504 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr"] Oct 29 22:31:48.910794 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:31:48.910768 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbe2a41_e6cc_4f1f_bfe0_d77a9ba7958a.slice/crio-8f624f2b0758d5d96e812354dc055d4f6267a95dd04b4098b67df2ec21fb469d WatchSource:0}: Error finding container 8f624f2b0758d5d96e812354dc055d4f6267a95dd04b4098b67df2ec21fb469d: Status 404 returned error can't find the container with id 8f624f2b0758d5d96e812354dc055d4f6267a95dd04b4098b67df2ec21fb469d Oct 29 22:31:48.913208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.913178 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:48.913285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.913240 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:48.913285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:48.913278 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:49.583116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:49.583080 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" event={"ID":"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a","Type":"ContainerStarted","Data":"42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f"} Oct 29 22:31:49.583577 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:49.583136 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" event={"ID":"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a","Type":"ContainerStarted","Data":"8f624f2b0758d5d96e812354dc055d4f6267a95dd04b4098b67df2ec21fb469d"} Oct 29 22:31:49.588460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:49.588440 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:49.590115 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:49.590083 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:31:49.590223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:49.590132 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 29 22:31:49.631174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:49.630739 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podStartSLOduration=1.6307215689999999 podStartE2EDuration="1.630721569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:49.628647982 +0000 UTC m=+1024.944042100" watchObservedRunningTime="2026-10-29 22:31:49.630721569 +0000 UTC m=+1024.946115650" Oct 29 22:31:50.589288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:50.589254 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:31:50.589721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:50.589314 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 29 22:31:51.588913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:51.588876 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:31:51.589106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:51.588937 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 29 22:31:52.589331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:52.589296 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:31:52.589829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:52.589356 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 29 22:31:53.589445 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:53.589413 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:31:53.589953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:53.589471 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 29 22:31:54.592669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:54.592467 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:31:54.593243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:54.593194 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 29 22:31:55.935614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:55.935584 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:55.936722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:55.936452 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:55.937435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:55.937413 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 22:31:59.766310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:31:59.766277 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:32:03.646770 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.646735 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n"] Oct 29 22:32:03.651765 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.651739 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.687710 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.687683 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n"] Oct 29 22:32:03.753754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.753719 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.753940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.753773 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.753940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.753798 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.753940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.753919 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxbf\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.754131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.753972 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.754131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.753998 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.754131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.754054 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.754131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.754093 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.754400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.754139 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.829063 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.829039 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7"] Oct 29 22:32:03.833084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.833058 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.855051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.855022 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.855288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.855268 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.855495 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.855478 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.855760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.855738 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856014 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7"] Oct 29 22:32:03.856101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856060 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856511 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856575 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856654 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856742 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856716 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856801 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856750 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856801 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856776 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.856894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.856856 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxbf\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.857317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.857296 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.857880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.857851 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.860914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.860890 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.861572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.861538 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.876087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.876027 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.877687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.877596 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxbf\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:03.958135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.958108 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.958314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.958169 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.958378 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.958351 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.958539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.958482 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.970181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.959546 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.970181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.959655 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.970181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.959749 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qn7\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.970181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.959970 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.970181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.960092 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:03.973243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:03.973218 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:04.064597 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064543 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.064745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064617 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.064745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064670 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.064745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064734 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.064927 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064819 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.064927 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064879 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.064927 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064906 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qn7\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.065102 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064938 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.065102 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.064974 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.065447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.065426 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.066184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.066123 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.066637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.066614 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.066862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.066843 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.067066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.067046 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.069465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.069438 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.070375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.070351 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.085759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.085732 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.086135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.086116 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qn7\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.148876 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.148825 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:04.303468 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.303442 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n"] Oct 29 22:32:04.304594 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:04.304568 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c27486_7961_4706_afb7_2e0731289faf.slice/crio-9ff38c890a3681b87f656ea0c23d5bcce79bd43ce45c78e907787b345d614b5e WatchSource:0}: Error finding container 9ff38c890a3681b87f656ea0c23d5bcce79bd43ce45c78e907787b345d614b5e: Status 404 returned error can't find the container with id 9ff38c890a3681b87f656ea0c23d5bcce79bd43ce45c78e907787b345d614b5e Oct 29 22:32:04.309570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.309539 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:04.309632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.309604 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:04.309673 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.309636 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:04.463507 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.463427 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7"] Oct 29 22:32:04.468393 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:04.468356 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bf2ed8_0ed0_4b75_9fa1_6dab7cc3ee0c.slice/crio-2e65e2e7ef72ef4141b9c55351ded803b74028182ae3c5c0e62823154f4b83b1 WatchSource:0}: Error finding container 2e65e2e7ef72ef4141b9c55351ded803b74028182ae3c5c0e62823154f4b83b1: Status 404 returned error can't find the container with id 2e65e2e7ef72ef4141b9c55351ded803b74028182ae3c5c0e62823154f4b83b1 Oct 29 22:32:04.471471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.471400 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:04.471536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.471482 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:04.471594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.471532 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:04.738489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.738386 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" event={"ID":"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c","Type":"ContainerStarted","Data":"f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109"} Oct 29 22:32:04.738906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.738477 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" event={"ID":"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c","Type":"ContainerStarted","Data":"2e65e2e7ef72ef4141b9c55351ded803b74028182ae3c5c0e62823154f4b83b1"} Oct 29 22:32:04.740301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.740275 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" event={"ID":"15c27486-7961-4706-afb7-2e0731289faf","Type":"ContainerStarted","Data":"5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e"} Oct 29 22:32:04.740408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.740321 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" event={"ID":"15c27486-7961-4706-afb7-2e0731289faf","Type":"ContainerStarted","Data":"9ff38c890a3681b87f656ea0c23d5bcce79bd43ce45c78e907787b345d614b5e"} Oct 29 22:32:04.785785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.785736 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podStartSLOduration=1.785722586 podStartE2EDuration="1.785722586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:04.783247587 +0000 UTC m=+1040.098641671" watchObservedRunningTime="2026-10-29 22:32:04.785722586 +0000 UTC m=+1040.101116747" Oct 29 22:32:04.829505 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.829454 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podStartSLOduration=1.82944106 podStartE2EDuration="1.82944106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:04.826508385 +0000 UTC m=+1040.141902461" watchObservedRunningTime="2026-10-29 22:32:04.82944106 +0000 UTC m=+1040.144835140" Oct 29 22:32:04.973738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:04.973702 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:05.149899 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:05.149792 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:05.151377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:05.151350 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:05.151507 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:05.151397 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:05.975430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:05.975396 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:05.975880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:05.975460 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:32:06.160178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:06.152744 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.160178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:06.152827 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:06.979322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:06.979239 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.979322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:06.979297 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:32:07.153177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:07.149490 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.153177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:07.149545 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:07.978199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:07.977517 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.978199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:07.977582 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:32:08.159176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:08.151073 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.171173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:08.151132 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:08.975525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:08.975487 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.975802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:08.975776 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:32:09.159172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:09.152507 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.159172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:09.152580 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:09.284017 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:32:09.283850 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:32:09.976283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:09.976247 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.976451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:09.976308 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:32:10.158176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:10.158114 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.158381 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:10.158205 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:10.973974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:10.973604 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.973974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:10.973658 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:32:11.149746 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:11.149710 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:11.149925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:11.149769 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:32:12.049457 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:12.049423 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:12.050502 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:12.050475 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:12.052107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:12.052079 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:32:12.312354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:12.312272 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:12.313704 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:12.313676 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:12.317048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:12.317025 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 22:32:13.759528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.759497 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k"] Oct 29 22:32:13.765101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.765077 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.782501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.782474 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k"] Oct 29 22:32:13.782733 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.782520 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.782880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.782840 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.782945 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.782887 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.782945 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.782920 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.783061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.783037 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.783116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.783076 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgps6\" (UniqueName: \"kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.790832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.790813 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a0287\"/\"distinct-2-w4-4a0287-kserve-self-signed-certs\"" Oct 29 22:32:13.883726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.883695 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884133 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884214 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884375 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884435 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgps6\" (UniqueName: \"kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884506 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884542 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.884874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.884835 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.885697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.885672 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.887725 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.887701 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.887805 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.887721 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:13.909192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:13.908406 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgps6\" (UniqueName: \"kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6\") pod \"distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:14.094014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:14.093923 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:14.439657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:14.439631 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k"] Oct 29 22:32:14.440041 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:14.440011 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c21edbe_94a0_43f4_80c8_49d95cc292b5.slice/crio-e92e6450331584b6828216c928f7f9367ea0097ebbd2e8d34f36ee98902617a9 WatchSource:0}: Error finding container e92e6450331584b6828216c928f7f9367ea0097ebbd2e8d34f36ee98902617a9: Status 404 returned error can't find the container with id e92e6450331584b6828216c928f7f9367ea0097ebbd2e8d34f36ee98902617a9 Oct 29 22:32:14.837833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:14.837798 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" event={"ID":"6c21edbe-94a0-43f4-80c8-49d95cc292b5","Type":"ContainerStarted","Data":"c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0"} Oct 29 22:32:14.837833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:14.837838 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" event={"ID":"6c21edbe-94a0-43f4-80c8-49d95cc292b5","Type":"ContainerStarted","Data":"e92e6450331584b6828216c928f7f9367ea0097ebbd2e8d34f36ee98902617a9"} Oct 29 22:32:14.884217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:14.884143 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podStartSLOduration=1.884126133 podStartE2EDuration="1.884126133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:14.880064085 +0000 UTC m=+1050.195458167" watchObservedRunningTime="2026-10-29 22:32:14.884126133 +0000 UTC m=+1050.199520215" Oct 29 22:32:15.094389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:15.094311 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:18.390243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.390208 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf"] Oct 29 22:32:18.394750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.394727 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.442844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.442815 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf"] Oct 29 22:32:18.446125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446095 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446388 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446571 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446674 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446711 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpn6c\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446741 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446939 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446766 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446939 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446818 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.446939 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.446864 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.459993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.459963 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg"] Oct 29 22:32:18.464989 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.464968 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.492463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.492439 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg"] Oct 29 22:32:18.547849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.547826 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.547982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.547897 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.547982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.547963 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.547983 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548011 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548039 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548065 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548238 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548272 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548304 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548319 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548346 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548385 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548422 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpn6c\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548450 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548465 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548511 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548564 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548595 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548660 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.548802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548774 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.548802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.548777 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.549244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.549221 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.550764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.550744 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.551069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.551050 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.568414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.568386 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.568822 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.568800 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpn6c\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.649739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.649672 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.649739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.649725 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.649920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.649836 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.649920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.649882 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.649909 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650024 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650054 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650173 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650228 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650496 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650437 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650541 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650788 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650766 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.650917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650893 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.651009 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.650982 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.652589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.652567 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.652992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.652974 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.671081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.671058 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.671268 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.671247 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.678958 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.678935 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2"] Oct 29 22:32:18.685108 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.685090 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.700892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.700872 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2"] Oct 29 22:32:18.709516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.709493 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:18.752924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.752882 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753097 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753352 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753542 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753524 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753600 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrd9\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753631 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753737 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753722 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753768 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.753844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.753798 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.785623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.785599 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:18.868248 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.866887 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.869278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.869245 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.869842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.869649 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.869965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.869794 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.870021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.869996 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrd9\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.870086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.870061 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.870470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.870444 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.870957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.870768 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.871040 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.870988 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.871798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.871725 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.872258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.872234 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.872361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.872273 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.872428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.872380 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.872624 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.872578 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.872728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.872713 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.872982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.872961 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.891948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.891922 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrd9\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:18.893773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:18.893648 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:19.001782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.001752 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:19.040632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.040602 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf"] Oct 29 22:32:19.041347 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:19.041321 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78b01c6_1ec2_4378_84c9_c57a6306b103.slice/crio-c8f136790e0be417634ac60a728cb72c9c35b404a289ecf06271f0ec24b0bc27 WatchSource:0}: Error finding container c8f136790e0be417634ac60a728cb72c9c35b404a289ecf06271f0ec24b0bc27: Status 404 returned error can't find the container with id c8f136790e0be417634ac60a728cb72c9c35b404a289ecf06271f0ec24b0bc27 Oct 29 22:32:19.043793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.043761 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.043892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.043838 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.043949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.043887 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.134316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.134210 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl"] Oct 29 22:32:19.139872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.139842 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.152382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.151355 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg"] Oct 29 22:32:19.160602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.160372 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl"] Oct 29 22:32:19.163017 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.162899 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.163223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.163197 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.163409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.163388 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.278634 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.278581 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.278832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.278814 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.278907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.278870 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.279006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.278981 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.279047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.279027 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxt24\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.279205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.279140 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.279294 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.279247 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.279333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.279321 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.279506 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.279483 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.324024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.323990 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2"] Oct 29 22:32:19.329011 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:19.328970 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5e7540_5da4_48b2_8316_20b92125aa78.slice/crio-a7cd4e20361f0a41b7c189dd4035bd7c002dcfb9976e6ac7612668c99e43c9b8 WatchSource:0}: Error finding container a7cd4e20361f0a41b7c189dd4035bd7c002dcfb9976e6ac7612668c99e43c9b8: Status 404 returned error can't find the container with id a7cd4e20361f0a41b7c189dd4035bd7c002dcfb9976e6ac7612668c99e43c9b8 Oct 29 22:32:19.331838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.331804 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.331932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.331875 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.331932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.331904 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.381307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381277 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381412 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381506 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381649 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381730 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381797 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381834 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381846 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381886 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.381973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381931 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxt24\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.382134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.381981 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.382134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.382015 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.382134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.382089 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.382768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.382744 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.391430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.391403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.391969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.391765 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.403359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.403336 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.404240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.404214 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxt24\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.469247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.469203 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:19.808261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.800646 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl"] Oct 29 22:32:19.877216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.877179 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.877334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.877267 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.877334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.877312 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:19.879116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.879093 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" event={"ID":"fec36d33-ee32-4d02-b608-3170bff96d4b","Type":"ContainerStarted","Data":"07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e"} Oct 29 22:32:19.879209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.879144 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" event={"ID":"fec36d33-ee32-4d02-b608-3170bff96d4b","Type":"ContainerStarted","Data":"99509d3958bdf3ac554f201add3be9cfda98033bd5ee205d7fc353725b23bc1e"} Oct 29 22:32:19.882292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.882267 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" event={"ID":"e78b01c6-1ec2-4378-84c9-c57a6306b103","Type":"ContainerStarted","Data":"1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489"} Oct 29 22:32:19.882480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.882455 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" event={"ID":"e78b01c6-1ec2-4378-84c9-c57a6306b103","Type":"ContainerStarted","Data":"c8f136790e0be417634ac60a728cb72c9c35b404a289ecf06271f0ec24b0bc27"} Oct 29 22:32:19.885299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.885278 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" event={"ID":"5c5e7540-5da4-48b2-8316-20b92125aa78","Type":"ContainerStarted","Data":"db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804"} Oct 29 22:32:19.885385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.885325 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" event={"ID":"5c5e7540-5da4-48b2-8316-20b92125aa78","Type":"ContainerStarted","Data":"a7cd4e20361f0a41b7c189dd4035bd7c002dcfb9976e6ac7612668c99e43c9b8"} Oct 29 22:32:19.886976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:19.886951 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" event={"ID":"d2f24af6-6096-4896-8ddd-e0083eee2132","Type":"ContainerStarted","Data":"d985e15ba91b9bc28738f34c5ac46a56fb0b6c9e83c0eaffad3e4335935b9608"} Oct 29 22:32:20.007243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.004524 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:20.007243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.004668 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.007243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.004716 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:20.031173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.030949 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podStartSLOduration=2.03092741 podStartE2EDuration="2.03092741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:19.972443752 +0000 UTC m=+1055.287837860" watchObservedRunningTime="2026-10-29 22:32:20.03092741 +0000 UTC m=+1055.346321492" Oct 29 22:32:20.051685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.035062 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podStartSLOduration=2.035041565 podStartE2EDuration="2.035041565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:20.030553863 +0000 UTC m=+1055.345947944" watchObservedRunningTime="2026-10-29 22:32:20.035041565 +0000 UTC m=+1055.350435646" Oct 29 22:32:20.083376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.083257 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podStartSLOduration=2.083234892 podStartE2EDuration="2.083234892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:20.081756968 +0000 UTC m=+1055.397151061" watchObservedRunningTime="2026-10-29 22:32:20.083234892 +0000 UTC m=+1055.398628973" Oct 29 22:32:20.721113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.721080 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:20.721834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.721807 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.722049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.722021 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:20.790451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.790394 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:20.792572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.792544 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.792823 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.792797 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:20.904357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:20.904317 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" event={"ID":"d2f24af6-6096-4896-8ddd-e0083eee2132","Type":"ContainerStarted","Data":"127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797"} Oct 29 22:32:21.006231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.005552 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.006231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.005607 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:21.545186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.513208 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:21.555192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.554303 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.555192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.554387 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:21.714639 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.714597 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.714956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.714924 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:21.788857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.788824 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.789456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.789426 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:21.948241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:21.945593 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:32:22.005673 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.005639 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.005980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.005940 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:22.010321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.010271 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.038327 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.038285 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.054350 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.054258 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.228175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.227270 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.258342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.258265 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.259019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.258974 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:32:22.390383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.390321 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:22.472702 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.472655 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.473028 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.472970 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:22.512257 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.512122 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.710175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.709984 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.710175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.710044 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:22.786657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.786572 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.786934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.786903 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:22.992184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.989256 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.992184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:22.989682 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:32:23.002824 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.002717 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.002824 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.002771 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:23.025070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.024603 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.028002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.027975 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:32:23.052739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.051254 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 29 22:32:23.053526 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.053505 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:32:23.064175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.055999 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podStartSLOduration=4.055983219 podStartE2EDuration="4.055983219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:20.964526113 +0000 UTC m=+1056.279920217" watchObservedRunningTime="2026-10-29 22:32:23.055983219 +0000 UTC m=+1058.371377301" Oct 29 22:32:23.066434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.066083 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.067001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.066672 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:32:23.131354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.124211 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.131354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.124318 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:32:23.258175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.251119 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.258175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.251567 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:32:23.272176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.258672 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.272176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.259039 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:32:23.272176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.259723 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.272176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.260023 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:32:23.394228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.393237 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.394228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.393842 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:32:23.423173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.422390 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:32:23.449175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.433664 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:32:23.474177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.470412 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.474177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.470468 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:23.553176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.540982 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.726183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.726041 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.726183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.726102 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:23.813084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.812448 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.813084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.812506 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:23.993916 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:23.993784 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 29 22:32:24.079284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.003380 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:24.079284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.016476 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.079284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.016533 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:24.079284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.045587 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:32:24.079284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.053369 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 29 22:32:24.079284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.074389 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:24.098429 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.098389 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:24.135176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.130716 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:24.135176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.131434 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:32:24.135176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.132843 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:32:24.135176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.132924 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:32:24.158643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.152128 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:32:24.430231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.399261 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:32:24.574342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.535248 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.575081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.575048 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:24.583363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.582273 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:24.665178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.583586 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:32:24.724173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.723864 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:32:24.730175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.725228 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.730175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.725281 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:24.849176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.848294 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:24.865285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:24.850514 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:32:25.077944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.077828 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.080109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.080074 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:32:25.130623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.104826 5073 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.138291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.106268 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.138558 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.138528 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:25.180269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.179930 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.183234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.183198 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:32:25.289979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.279216 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.349260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.320462 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.349260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.320577 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:25.553464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.553413 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.565321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.565291 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.565611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.565567 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:25.586320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.586282 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:25.638992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.638888 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.801134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.801086 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.802252 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.964404 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.850410 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.853268 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.832323 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.850341 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:25.964508 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:25.993233 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:32:25.991795 5073 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.141s" Oct 29 22:32:26.054176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.045759 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.054176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.045817 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:26.095901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.095861 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.105654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.103377 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.214880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.214839 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.228174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.215507 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:32:26.255934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.255845 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:32:26.303684 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.280253 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.484762 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.484727 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.529412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.485018 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:26.567685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.567643 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.621573 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.590795 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:26.622094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.622069 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:32:26.622393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.622377 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:32:26.665184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.645223 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:32:26.726177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.709837 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.726177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.709900 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:26.800283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.800090 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.800283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.800144 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:26.885172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.841883 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:32:26.887685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.847640 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.993696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.993658 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:32:26.995898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:26.995870 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:32:27.040619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.040581 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:27.095049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.046660 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:27.095049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.046711 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:27.169824 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.169792 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:32:27.170598 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.170577 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:32:27.170795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.170781 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:32:27.171143 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.171117 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:32:27.171607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.171512 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:32:27.172273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.172252 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:32:27.172368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.172302 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:32:27.172368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.172338 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:32:27.172368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.172354 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:32:27.172368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.172368 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:32:27.197280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.197257 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:32:27.201135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.201082 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:32:27.470397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.470365 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:27.470686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.470658 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:27.735224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.734661 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:27.735224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.734715 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 29 22:32:27.864187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.859413 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:27.864187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:27.859493 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:32:28.002324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.002234 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.002324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.002297 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 22:32:28.470098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.470066 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.470738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.470696 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:32:28.968891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.968864 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:28.970397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.970361 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:28.972731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:28.972200 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 22:32:29.024972 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.024939 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:29.026266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.025673 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:29.026779 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.026759 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:32:29.066123 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.066093 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:29.066626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.066540 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:29.067732 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.067700 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 22:32:29.495543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.495507 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms"] Oct 29 22:32:29.500852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.500826 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.516797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.516716 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:29.516952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.516902 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:29.517997 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.517972 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms"] Oct 29 22:32:29.669480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.669440 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.669659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.669590 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4c7p\" (UniqueName: \"kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.669659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.669652 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.717681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.716928 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:29.718084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.718034 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:29.719410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.719321 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 22:32:29.771713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.771637 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.771896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.771778 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4c7p\" (UniqueName: \"kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.771896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.771814 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.772338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.772314 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.774423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.774391 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.797630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.797606 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4c7p\" (UniqueName: \"kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p\") pod \"maas-api-revoke-sub-8b2ce7be-qmwms\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:29.821745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:29.821709 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:30.147775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.147748 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms"] Oct 29 22:32:30.148603 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:30.148581 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e9f767_ead9_4d0c_a17e_29462ca243c6.slice/crio-9adbf95acaf797177fc048108c2efb5763139e636b3656839b03b987120fe448 WatchSource:0}: Error finding container 9adbf95acaf797177fc048108c2efb5763139e636b3656839b03b987120fe448: Status 404 returned error can't find the container with id 9adbf95acaf797177fc048108c2efb5763139e636b3656839b03b987120fe448 Oct 29 22:32:30.873073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.873043 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc"] Oct 29 22:32:30.881330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.881302 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:30.904843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.904814 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc"] Oct 29 22:32:30.912663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.912642 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-95672c\"/\"distinct-2-w3-95672c-kserve-self-signed-certs\"" Oct 29 22:32:30.990116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.990086 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6c2\" (UniqueName: \"kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:30.990277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.990126 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:30.990325 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.990294 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:30.990519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.990495 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:30.990697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.990677 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:30.990739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:30.990725 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.093254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093219 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6c2\" (UniqueName: \"kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.093428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093278 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.093428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093382 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.093544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093515 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.094067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093659 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.094067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093687 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.094067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.094067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.093968 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.094067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.094023 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.096217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.096192 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.096217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.096212 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.113931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.113903 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6c2\" (UniqueName: \"kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2\") pod \"distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.161425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.159851 5073 generic.go:356] "Generic (PLEG): container finished" podID="a4e9f767-ead9-4d0c-a17e-29462ca243c6" containerID="50d708a5e30e37dcaef9f55912e168167d1ff20828a96e24a64427cedaf64297" exitCode=0 Oct 29 22:32:31.161425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.161267 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" event={"ID":"a4e9f767-ead9-4d0c-a17e-29462ca243c6","Type":"ContainerDied","Data":"50d708a5e30e37dcaef9f55912e168167d1ff20828a96e24a64427cedaf64297"} Oct 29 22:32:31.161425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.161320 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" event={"ID":"a4e9f767-ead9-4d0c-a17e-29462ca243c6","Type":"ContainerStarted","Data":"9adbf95acaf797177fc048108c2efb5763139e636b3656839b03b987120fe448"} Oct 29 22:32:31.216550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.216513 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:31.551293 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:31.551262 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaec2da7_e3b0_4000_adc5_bba4afca80a1.slice/crio-2e9a612abff4c71f37e0c985fdde82fe176568a56824ed650397671388b8c424 WatchSource:0}: Error finding container 2e9a612abff4c71f37e0c985fdde82fe176568a56824ed650397671388b8c424: Status 404 returned error can't find the container with id 2e9a612abff4c71f37e0c985fdde82fe176568a56824ed650397671388b8c424 Oct 29 22:32:31.551970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:31.551925 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc"] Oct 29 22:32:32.168617 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.168578 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" event={"ID":"aaec2da7-e3b0-4000-adc5-bba4afca80a1","Type":"ContainerStarted","Data":"1a670d7b0355ae01d4697635036f3aabf628cb265963fd2e33618c1eaa88d7b0"} Oct 29 22:32:32.169086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.168630 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" event={"ID":"aaec2da7-e3b0-4000-adc5-bba4afca80a1","Type":"ContainerStarted","Data":"2e9a612abff4c71f37e0c985fdde82fe176568a56824ed650397671388b8c424"} Oct 29 22:32:32.219487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.219457 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:32.271165 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.271082 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podStartSLOduration=2.27106518 podStartE2EDuration="2.27106518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:32.264456868 +0000 UTC m=+1067.579850966" watchObservedRunningTime="2026-10-29 22:32:32.27106518 +0000 UTC m=+1067.586459261" Oct 29 22:32:32.328980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.328958 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:32.439028 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.438939 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca\") pod \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " Oct 29 22:32:32.439229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.439092 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca\") pod \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " Oct 29 22:32:32.439229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.439132 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p\" (UniqueName: \"kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p\") pod \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\" (UID: \"a4e9f767-ead9-4d0c-a17e-29462ca243c6\") " Oct 29 22:32:32.439729 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.439699 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca" pod "a4e9f767-ead9-4d0c-a17e-29462ca243c6" (UID: "a4e9f767-ead9-4d0c-a17e-29462ca243c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:32.439940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.439922 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:32:32.442136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.442101 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca" pod "a4e9f767-ead9-4d0c-a17e-29462ca243c6" (UID: "a4e9f767-ead9-4d0c-a17e-29462ca243c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:32.444325 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.444297 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p" pod "a4e9f767-ead9-4d0c-a17e-29462ca243c6" (UID: "a4e9f767-ead9-4d0c-a17e-29462ca243c6"). InnerVolumeSpecName "kube-api-access-g4c7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:32.541239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.541204 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9f767-ead9-4d0c-a17e-29462ca243c6-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:32:32.541239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:32.541238 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4c7p\" (UniqueName: \"kubernetes.io/projected/a4e9f767-ead9-4d0c-a17e-29462ca243c6-kube-api-access-g4c7p\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:32:33.181306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:33.181221 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" Oct 29 22:32:33.182626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:33.181806 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms" event={"ID":"a4e9f767-ead9-4d0c-a17e-29462ca243c6","Type":"ContainerDied","Data":"9adbf95acaf797177fc048108c2efb5763139e636b3656839b03b987120fe448"} Oct 29 22:32:33.183792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:33.183775 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9adbf95acaf797177fc048108c2efb5763139e636b3656839b03b987120fe448" Oct 29 22:32:34.769334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:34.769290 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.815336 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:34.815294 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.862067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:34.862025 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.874549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:34.874514 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.019275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.019232 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.048351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.048255 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.063255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.063216 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.117264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.117223 5073 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.181272 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.181227 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.259285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.259242 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.775293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.775245 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.880547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.875240 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.889589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.881392 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:32:35.893170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.889837 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:32:35.900182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.875820 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.900182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.893930 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:32:35.900182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:35.895119 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.010066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.010015 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:32:36.029660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.029575 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:32:36.030801 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.030776 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:32:36.068822 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.063318 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.089264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.089236 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:32:36.193436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.193395 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.194455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.194414 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:32:36.284609 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.284534 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:32:36.293536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.293505 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:32:36.295497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.295446 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:32:36.391118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.391092 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:32:36.391760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:36.391742 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:32:39.255674 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:32:39.255635 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:32:41.219967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:41.219935 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:41.232918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:41.232891 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:41.352907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:41.351082 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:32:44.105584 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:44.105553 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:44.117000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:44.116975 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:32:48.479048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.479011 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t"] Oct 29 22:32:48.479942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.479921 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a4e9f767-ead9-4d0c-a17e-29462ca243c6" containerName="revoke-keys" Oct 29 22:32:48.479942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.479942 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e9f767-ead9-4d0c-a17e-29462ca243c6" containerName="revoke-keys" Oct 29 22:32:48.480124 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.480101 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e9f767-ead9-4d0c-a17e-29462ca243c6" containerName="revoke-keys" Oct 29 22:32:48.485386 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.485364 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.517751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.517717 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t"] Oct 29 22:32:48.559311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559268 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559403 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559472 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559530 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559556 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559779 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tl2x\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559813 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559857 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.559992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.559967 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661085 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661118 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661242 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tl2x\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661266 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661291 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661351 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661445 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661579 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661493 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661579 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661540 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661579 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661575 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.661871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661838 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.662095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661880 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.662095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.661904 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.662095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.662082 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.663677 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.663650 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.663993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.663971 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.682360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.682334 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.682480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.682415 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tl2x\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:48.799713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:48.799646 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:49.120200 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:49.120133 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4a3856_30ab_4409_8c5b_dc32b1c40d84.slice/crio-7893bef968a282af3dade9e321d95d197e374b6674a172152af750076ba7e4b4 WatchSource:0}: Error finding container 7893bef968a282af3dade9e321d95d197e374b6674a172152af750076ba7e4b4: Status 404 returned error can't find the container with id 7893bef968a282af3dade9e321d95d197e374b6674a172152af750076ba7e4b4 Oct 29 22:32:49.120356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.120304 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t"] Oct 29 22:32:49.122966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.122931 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:49.123090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.123006 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:49.123090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.123050 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:49.152057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.152026 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc"] Oct 29 22:32:49.158305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.158283 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.189055 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.189029 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc"] Oct 29 22:32:49.269564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.269531 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.269721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.269665 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.269792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.269728 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.269871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.269831 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.269926 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.269873 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.270019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.270001 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.270135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.270110 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5776t\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.270288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.270260 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.270356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.270326 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371505 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371539 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5776t\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371579 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371629 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371689 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371774 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371837 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371880 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.371993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.371905 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.372239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.372186 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.372335 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.372257 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.372428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.372406 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.372683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.372575 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.372938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.372913 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.374340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.374315 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.374460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.374444 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.384236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.384209 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" event={"ID":"2c4a3856-30ab-4409-8c5b-dc32b1c40d84","Type":"ContainerStarted","Data":"3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c"} Oct 29 22:32:49.384334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.384267 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" event={"ID":"2c4a3856-30ab-4409-8c5b-dc32b1c40d84","Type":"ContainerStarted","Data":"7893bef968a282af3dade9e321d95d197e374b6674a172152af750076ba7e4b4"} Oct 29 22:32:49.394618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.394594 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.406969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.406946 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5776t\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.447402 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.447356 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podStartSLOduration=1.447345024 podStartE2EDuration="1.447345024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:49.444219327 +0000 UTC m=+1084.759613408" watchObservedRunningTime="2026-10-29 22:32:49.447345024 +0000 UTC m=+1084.762739105" Oct 29 22:32:49.486315 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.486289 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:49.511586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.511555 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp"] Oct 29 22:32:49.517198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.517143 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.548043 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.548016 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:49.548753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.548728 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:32:49.566870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.566837 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp"] Oct 29 22:32:49.579462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.577684 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.579462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.577805 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.579462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.577935 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fqw\" (UniqueName: \"kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.685350 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.682294 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fqw\" (UniqueName: \"kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.685350 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.682611 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.685350 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.682715 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.686204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.686073 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.692197 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.689016 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.707837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.707798 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fqw\" (UniqueName: \"kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw\") pod \"maas-api-revoke-sub-8e7b24aa-z29zp\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:49.800328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.800296 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:49.845405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.845374 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc"] Oct 29 22:32:49.848344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.848310 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:49.848433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.848379 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:49.848433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.848407 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:32:49.848705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:49.848684 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:50.187179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.187140 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp"] Oct 29 22:32:50.187647 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:32:50.187621 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6a3fb1_f635_4eb3_ae36_29389a227fe8.slice/crio-18393f1a2bb45ce09499a71570e023e2c7187625fe06643f22ed7922b98b8fab WatchSource:0}: Error finding container 18393f1a2bb45ce09499a71570e023e2c7187625fe06643f22ed7922b98b8fab: Status 404 returned error can't find the container with id 18393f1a2bb45ce09499a71570e023e2c7187625fe06643f22ed7922b98b8fab Oct 29 22:32:50.401437 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.401397 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" event={"ID":"fe6a3fb1-f635-4eb3-ae36-29389a227fe8","Type":"ContainerStarted","Data":"18393f1a2bb45ce09499a71570e023e2c7187625fe06643f22ed7922b98b8fab"} Oct 29 22:32:50.403628 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.403594 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" event={"ID":"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0","Type":"ContainerStarted","Data":"22a42d5c5fb57c41a02a13fa17c78fc6f859a0639a9060ab55b0b094d15da3cc"} Oct 29 22:32:50.403754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.403647 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" event={"ID":"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0","Type":"ContainerStarted","Data":"ba2bf53aa158f2359ed7ca1995f16fbdef926e9f2d763aa96dbb9103aaf5e7c7"} Oct 29 22:32:50.455126 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.455027 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podStartSLOduration=1.455007345 podStartE2EDuration="1.455007345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:50.450549016 +0000 UTC m=+1085.765943092" watchObservedRunningTime="2026-10-29 22:32:50.455007345 +0000 UTC m=+1085.770401426" Oct 29 22:32:50.487304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.487265 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:50.490874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.489963 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:50.490874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.490020 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:50.801685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.801653 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:32:50.801843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:50.801715 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:32:51.434176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:51.433717 5073 generic.go:356] "Generic (PLEG): container finished" podID="fe6a3fb1-f635-4eb3-ae36-29389a227fe8" containerID="3f55039a7fc6b7b3efc11880e3571d8e1f0e2563e7cdd58e22a8b73db9ce9d1e" exitCode=0 Oct 29 22:32:51.438176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:51.435215 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" event={"ID":"fe6a3fb1-f635-4eb3-ae36-29389a227fe8","Type":"ContainerDied","Data":"3f55039a7fc6b7b3efc11880e3571d8e1f0e2563e7cdd58e22a8b73db9ce9d1e"} Oct 29 22:32:51.486640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:51.486606 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.486800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:51.486654 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:51.800289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:51.800260 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.800765 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:51.800742 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:32:52.486936 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:52.486890 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.487093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:52.486947 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:52.800289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:52.800202 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.800289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:52.800250 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:32:52.900588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:52.900563 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:53.088470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.088366 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca\") pod \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " Oct 29 22:32:53.088658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.088627 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw\" (UniqueName: \"kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw\") pod \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " Oct 29 22:32:53.088767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.088751 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca\") pod \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\" (UID: \"fe6a3fb1-f635-4eb3-ae36-29389a227fe8\") " Oct 29 22:32:53.088875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.088849 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca" pod "fe6a3fb1-f635-4eb3-ae36-29389a227fe8" (UID: "fe6a3fb1-f635-4eb3-ae36-29389a227fe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:53.089387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.089367 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:32:53.091026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.091005 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw" pod "fe6a3fb1-f635-4eb3-ae36-29389a227fe8" (UID: "fe6a3fb1-f635-4eb3-ae36-29389a227fe8"). InnerVolumeSpecName "kube-api-access-26fqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:53.091385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.091332 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca" pod "fe6a3fb1-f635-4eb3-ae36-29389a227fe8" (UID: "fe6a3fb1-f635-4eb3-ae36-29389a227fe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:53.191532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.191504 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:32:53.191532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.191528 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fqw\" (UniqueName: \"kubernetes.io/projected/fe6a3fb1-f635-4eb3-ae36-29389a227fe8-kube-api-access-26fqw\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:32:53.479683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.479642 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" event={"ID":"fe6a3fb1-f635-4eb3-ae36-29389a227fe8","Type":"ContainerDied","Data":"18393f1a2bb45ce09499a71570e023e2c7187625fe06643f22ed7922b98b8fab"} Oct 29 22:32:53.479961 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.479947 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp" Oct 29 22:32:53.480014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.479993 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18393f1a2bb45ce09499a71570e023e2c7187625fe06643f22ed7922b98b8fab" Oct 29 22:32:53.487331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.487310 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.487501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.487479 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:53.800911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.800834 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.800911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:53.800883 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:32:54.487472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:54.487434 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.487683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:54.487494 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:54.801001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:54.800925 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.801001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:54.800977 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:32:55.486865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:55.486834 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.487027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:55.486886 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:55.800551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:55.800473 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.800551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:55.800523 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:32:56.487062 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:56.487025 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:32:56.487464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:56.487091 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:32:56.808546 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:56.808459 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:56.809436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:56.809398 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:56.810131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:56.810112 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:32:57.537511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:57.537339 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:57.538431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:57.538407 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:32:57.539303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:32:57.539282 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 22:33:01.774817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:01.774785 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:33:04.660179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:04.657253 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:04.778178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:04.777231 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 29 22:33:04.883353 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:04.883300 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:33:05.015175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.014237 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:33:05.053293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.053244 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.059270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.059228 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.082281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.082235 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.105818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.105773 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 29 22:33:05.111277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.111235 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.314462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.297549 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 29 22:33:05.315669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.314866 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.316321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.316279 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.345298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.334628 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.394285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.357194 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.396817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.396408 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:33:05.605232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.599681 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.968234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.957721 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:33:05.968234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:05.958466 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:33:06.031506 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.025682 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.031506 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.027283 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:33:06.093310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.076882 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.093310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.087794 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:33:06.093310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.088578 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:33:06.093310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.088640 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:33:06.117924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.094419 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:06.117924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.094788 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:33:06.117924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.115319 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.117924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.115598 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:33:06.348613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.347293 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.434186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.357924 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:33:06.441564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.358202 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:33:06.441839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.441819 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:33:06.467208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.359292 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:06.507309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.398520 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.507582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.394794 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.508301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.508269 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:33:06.574658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.574622 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:33:06.631482 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.631379 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.636443 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:06.636416 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:33:07.271177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.261542 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.271177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.266331 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:33:07.271177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.267757 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.292491 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.292451 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 29 22:33:07.349194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.325230 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.539256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.473503 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.564348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.524823 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.605330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.605284 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:33:07.687178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.681258 5073 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-wsjxt container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.14:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:07.687178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.681334 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-wsjxt" podUID="01ed9874-d3cc-4b95-a198-348f68659495" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.687178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.684281 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.739177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.711974 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:33:07.818516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.713073 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:07.832247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.713503 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:33:07.836286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:07.753874 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:08.096890 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:08.096797 5073 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.023s" Oct 29 22:33:08.097212 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.097190 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:33:08.264320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.264272 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.295074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.295027 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.304189 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.296250 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:33:08.334825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.334787 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.599178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.568774 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.614736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.614244 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.730176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.714719 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.760728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.758620 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.760728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.759866 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 29 22:33:08.788014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.786672 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.788014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.786761 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:33:08.822329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.822277 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.823355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.822628 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.897266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.855947 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.897266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.856331 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.897266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:08.856615 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.265667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.265622 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.316177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.296899 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.317103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.305928 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.363416 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.357367 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.625988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.625889 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:33:09.672286 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:09.664330 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:33:09.719188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.714261 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.775231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.769797 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.775231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.770629 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:33:09.792831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.787840 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.793829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.793795 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.871392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.871347 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:09.904254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.896279 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:33:09.904254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.896321 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:09.988165 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:09.987914 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:33:10.217456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.217414 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.288244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.272252 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.385140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.385097 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:33:10.417201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.417140 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.441177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.418018 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:33:10.573228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.561939 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:33:10.662022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.661978 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.668636 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.668613 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:33:10.727362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.727325 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:33:10.778493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.778454 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:33:10.794441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.794396 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:10.801218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.795441 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.831203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.809434 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:33:10.966535 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:10.964431 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:11.033285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.033256 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:33:11.054953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.054899 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.101380 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.101279 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.209171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.194485 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:33:11.293172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.270753 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:33:11.374258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.357688 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 29 22:33:11.374519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.374202 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:33:11.374629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.374561 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:33:11.374629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.374599 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:33:11.423169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.415132 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:33:11.461552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.461519 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:33:11.472744 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.472720 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:33:11.474463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:11.474446 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:33:17.748055 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.748013 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px"] Oct 29 22:33:17.748883 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.748861 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fe6a3fb1-f635-4eb3-ae36-29389a227fe8" containerName="revoke-keys" Oct 29 22:33:17.748966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.748888 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6a3fb1-f635-4eb3-ae36-29389a227fe8" containerName="revoke-keys" Oct 29 22:33:17.749078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.749064 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6a3fb1-f635-4eb3-ae36-29389a227fe8" containerName="revoke-keys" Oct 29 22:33:17.755247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.755225 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.771611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.771585 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px"] Oct 29 22:33:17.781308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.781288 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:17.781397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.781374 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:17.786093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.786057 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647tx\" (UniqueName: \"kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.786223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.786161 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.786373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.786350 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.887191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.887113 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647tx\" (UniqueName: \"kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.887379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.887203 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.887379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.887348 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.888120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.888040 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.889875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.889842 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:17.908233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:17.908207 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647tx\" (UniqueName: \"kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx\") pod \"maas-api-revoke-sub-0d79736b-jx4px\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:18.072568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.072473 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:18.403969 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:33:18.403936 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e3b60a_f86b_4e90_9fca_986cbf6371ab.slice/crio-18e4912840590b21259ff29aecbf56e35ce2e3f91092c76184dd6ea9f5c35b21 WatchSource:0}: Error finding container 18e4912840590b21259ff29aecbf56e35ce2e3f91092c76184dd6ea9f5c35b21: Status 404 returned error can't find the container with id 18e4912840590b21259ff29aecbf56e35ce2e3f91092c76184dd6ea9f5c35b21 Oct 29 22:33:18.404699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.404676 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px"] Oct 29 22:33:18.676856 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.676786 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7"] Oct 29 22:33:18.682520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.682421 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.704681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.704658 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7"] Oct 29 22:33:18.803418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803381 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.803797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803505 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.803797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803544 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.803797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803561 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.803797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803580 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.803797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803652 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.803797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803772 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.804019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803813 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.804019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.803851 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztg9j\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.905696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.905668 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.905923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.905781 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.905923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.905836 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.905923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.905869 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.905923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.905900 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.905941 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906010 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906035 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906077 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906076 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztg9j\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906337 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906417 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906701 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.906914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.906896 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.908375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.908353 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.908832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.908810 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.926408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.926385 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.926563 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.926542 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztg9j\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:18.996342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:18.996321 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:19.321731 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:33:19.321701 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c80d264_862f_4f44_8732_576fc9d67a22.slice/crio-366723c66ccb2da11e4a9594376744737763b1c661042de7b8b987b55eb55969 WatchSource:0}: Error finding container 366723c66ccb2da11e4a9594376744737763b1c661042de7b8b987b55eb55969: Status 404 returned error can't find the container with id 366723c66ccb2da11e4a9594376744737763b1c661042de7b8b987b55eb55969 Oct 29 22:33:19.322456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.322432 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7"] Oct 29 22:33:19.324798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.324763 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:33:19.324892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.324831 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:33:19.324892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.324859 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:33:19.404494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.404464 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" event={"ID":"8c80d264-862f-4f44-8732-576fc9d67a22","Type":"ContainerStarted","Data":"366723c66ccb2da11e4a9594376744737763b1c661042de7b8b987b55eb55969"} Oct 29 22:33:19.406213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.406175 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" event={"ID":"b2e3b60a-f86b-4e90-9fca-986cbf6371ab","Type":"ContainerStarted","Data":"7f1a458e38ea0b83c649d7e3ca285022513668492b8624167d3d0239767abe86"} Oct 29 22:33:19.406317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.406229 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" event={"ID":"b2e3b60a-f86b-4e90-9fca-986cbf6371ab","Type":"ContainerStarted","Data":"18e4912840590b21259ff29aecbf56e35ce2e3f91092c76184dd6ea9f5c35b21"} Oct 29 22:33:19.452185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:19.452099 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" podStartSLOduration=2.452083189 podStartE2EDuration="2.452083189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:19.448750997 +0000 UTC m=+1114.764145077" watchObservedRunningTime="2026-10-29 22:33:19.452083189 +0000 UTC m=+1114.767477265" Oct 29 22:33:20.443101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.443061 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" event={"ID":"8c80d264-862f-4f44-8732-576fc9d67a22","Type":"ContainerStarted","Data":"d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee"} Oct 29 22:33:20.444819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.444792 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2e3b60a-f86b-4e90-9fca-986cbf6371ab" containerID="7f1a458e38ea0b83c649d7e3ca285022513668492b8624167d3d0239767abe86" exitCode=0 Oct 29 22:33:20.444962 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.444880 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" event={"ID":"b2e3b60a-f86b-4e90-9fca-986cbf6371ab","Type":"ContainerDied","Data":"7f1a458e38ea0b83c649d7e3ca285022513668492b8624167d3d0239767abe86"} Oct 29 22:33:20.534849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.534787 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podStartSLOduration=2.534767972 podStartE2EDuration="2.534767972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:20.502292882 +0000 UTC m=+1115.817686962" watchObservedRunningTime="2026-10-29 22:33:20.534767972 +0000 UTC m=+1115.850162054" Oct 29 22:33:20.996774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.996736 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:20.997930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.997906 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:33:20.998054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:20.997953 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:33:21.705339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.705315 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:21.867395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.867307 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca\") pod \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " Oct 29 22:33:21.867544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.867436 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca\") pod \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " Oct 29 22:33:21.867544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.867474 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx\" (UniqueName: \"kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx\") pod \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\" (UID: \"b2e3b60a-f86b-4e90-9fca-986cbf6371ab\") " Oct 29 22:33:21.867803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.867779 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca" pod "b2e3b60a-f86b-4e90-9fca-986cbf6371ab" (UID: "b2e3b60a-f86b-4e90-9fca-986cbf6371ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:21.868019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.868001 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:21.869697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.869674 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx" pod "b2e3b60a-f86b-4e90-9fca-986cbf6371ab" (UID: "b2e3b60a-f86b-4e90-9fca-986cbf6371ab"). InnerVolumeSpecName "kube-api-access-647tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:21.869789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.869714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca" pod "b2e3b60a-f86b-4e90-9fca-986cbf6371ab" (UID: "b2e3b60a-f86b-4e90-9fca-986cbf6371ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:21.969562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.969531 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:21.969562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.969559 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647tx\" (UniqueName: \"kubernetes.io/projected/b2e3b60a-f86b-4e90-9fca-986cbf6371ab-kube-api-access-647tx\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:21.996957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.996927 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:33:21.997095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:21.996974 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:33:22.466075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:22.466044 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" Oct 29 22:33:22.466288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:22.466080 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px" event={"ID":"b2e3b60a-f86b-4e90-9fca-986cbf6371ab","Type":"ContainerDied","Data":"18e4912840590b21259ff29aecbf56e35ce2e3f91092c76184dd6ea9f5c35b21"} Oct 29 22:33:22.466288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:22.466126 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e4912840590b21259ff29aecbf56e35ce2e3f91092c76184dd6ea9f5c35b21" Oct 29 22:33:22.997341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:22.997305 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:33:22.997756 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:22.997357 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:33:23.997343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:23.997311 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:33:23.997749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:23.997378 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:33:24.996880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:24.996839 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:33:24.997048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:24.996907 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:33:25.997378 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:25.997343 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:33:25.997888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:25.997397 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:33:27.000815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:27.000786 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:27.001315 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:27.001300 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:27.002085 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:27.002063 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:33:31.215336 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.215296 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc"] Oct 29 22:33:31.215911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.215898 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2e3b60a-f86b-4e90-9fca-986cbf6371ab" containerName="revoke-keys" Oct 29 22:33:31.215956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.215912 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e3b60a-f86b-4e90-9fca-986cbf6371ab" containerName="revoke-keys" Oct 29 22:33:31.216044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.216035 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e3b60a-f86b-4e90-9fca-986cbf6371ab" containerName="revoke-keys" Oct 29 22:33:31.219533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.219500 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.234949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.234918 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.235107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.235057 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.235301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.235280 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzz4g\" (UniqueName: \"kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.238474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.238453 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc"] Oct 29 22:33:31.247267 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.247246 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:31.247425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.247338 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:31.336084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.336057 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.336262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.336209 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzz4g\" (UniqueName: \"kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.336308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.336293 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.336715 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.336696 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.338780 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.338762 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.357649 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.357626 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzz4g\" (UniqueName: \"kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g\") pod \"maas-api-revoke-sub-3f2e7375-2z8bc\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.535378 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.535289 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:31.847763 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:33:31.847720 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06266413_4748_40d7_b64f_68b301ddbec4.slice/crio-5052f4c546bd05bb9a90573daa6c327697cee9a27c3e57fc8ef844e321c142ae WatchSource:0}: Error finding container 5052f4c546bd05bb9a90573daa6c327697cee9a27c3e57fc8ef844e321c142ae: Status 404 returned error can't find the container with id 5052f4c546bd05bb9a90573daa6c327697cee9a27c3e57fc8ef844e321c142ae Oct 29 22:33:31.847898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:31.847785 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc"] Oct 29 22:33:32.540286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:32.540251 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" event={"ID":"06266413-4748-40d7-b64f-68b301ddbec4","Type":"ContainerStarted","Data":"787ac5de1d1e2bce3a2884a601b00f46ae5f7e373e1ef7ea9d8860a4b612d677"} Oct 29 22:33:32.540718 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:32.540305 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" event={"ID":"06266413-4748-40d7-b64f-68b301ddbec4","Type":"ContainerStarted","Data":"5052f4c546bd05bb9a90573daa6c327697cee9a27c3e57fc8ef844e321c142ae"} Oct 29 22:33:32.582291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:32.582250 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" podStartSLOduration=1.582235191 podStartE2EDuration="1.582235191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:32.580418526 +0000 UTC m=+1127.895812607" watchObservedRunningTime="2026-10-29 22:33:32.582235191 +0000 UTC m=+1127.897629270" Oct 29 22:33:33.548544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:33.548507 5073 generic.go:356] "Generic (PLEG): container finished" podID="06266413-4748-40d7-b64f-68b301ddbec4" containerID="787ac5de1d1e2bce3a2884a601b00f46ae5f7e373e1ef7ea9d8860a4b612d677" exitCode=0 Oct 29 22:33:33.549056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:33.548581 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" event={"ID":"06266413-4748-40d7-b64f-68b301ddbec4","Type":"ContainerDied","Data":"787ac5de1d1e2bce3a2884a601b00f46ae5f7e373e1ef7ea9d8860a4b612d677"} Oct 29 22:33:34.758853 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.758827 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:34.885260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.885087 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca\") pod \"06266413-4748-40d7-b64f-68b301ddbec4\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " Oct 29 22:33:34.885260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.885162 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g\" (UniqueName: \"kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g\") pod \"06266413-4748-40d7-b64f-68b301ddbec4\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " Oct 29 22:33:34.885468 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.885393 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca\") pod \"06266413-4748-40d7-b64f-68b301ddbec4\" (UID: \"06266413-4748-40d7-b64f-68b301ddbec4\") " Oct 29 22:33:34.885767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.885743 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca" pod "06266413-4748-40d7-b64f-68b301ddbec4" (UID: "06266413-4748-40d7-b64f-68b301ddbec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:34.886331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.886304 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:34.887792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.887770 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca" pod "06266413-4748-40d7-b64f-68b301ddbec4" (UID: "06266413-4748-40d7-b64f-68b301ddbec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:34.887886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.887822 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g" pod "06266413-4748-40d7-b64f-68b301ddbec4" (UID: "06266413-4748-40d7-b64f-68b301ddbec4"). InnerVolumeSpecName "kube-api-access-vzz4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:34.987076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.987048 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06266413-4748-40d7-b64f-68b301ddbec4-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:34.987076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:34.987076 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzz4g\" (UniqueName: \"kubernetes.io/projected/06266413-4748-40d7-b64f-68b301ddbec4-kube-api-access-vzz4g\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:35.563907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:35.563874 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" Oct 29 22:33:35.564073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:35.563877 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc" event={"ID":"06266413-4748-40d7-b64f-68b301ddbec4","Type":"ContainerDied","Data":"5052f4c546bd05bb9a90573daa6c327697cee9a27c3e57fc8ef844e321c142ae"} Oct 29 22:33:35.564073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:35.563996 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5052f4c546bd05bb9a90573daa6c327697cee9a27c3e57fc8ef844e321c142ae" Oct 29 22:33:39.618281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.618226 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:33:39.883265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.791170 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:39.883265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.792972 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.883265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.816397 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.883265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.829224 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:33:39.883265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.829548 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.954091 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:39.909953 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:33:39.957392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.955884 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.990646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:39.990489 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.206504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.206018 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.282113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.281227 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.282113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.281883 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:33:40.284160 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.284104 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.814176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.810907 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.815703 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.833439 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.833802 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.842220 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.842548 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.842835 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:33:40.858243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.842888 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:33:40.999175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:40.998906 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.037184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.000358 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.128265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.037534 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.295755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.295714 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:33:41.324022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.296037 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.296105 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.307862 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.355517 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.355559 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.355594 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.355628 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.355664 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:33:41.382277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.358804 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.708423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.708380 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.919359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:41.919315 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:42.113566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.060123 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.113566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.060247 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.155797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.155753 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.163183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.161230 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:42.185289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.165228 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.312139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.194541 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:33:42.312536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.227687 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.387514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.386200 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.447240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.447199 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:33:42.464310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.464269 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:33:42.512184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.489723 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:33:42.622177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.618394 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.697305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.689304 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 29 22:33:42.697305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.689457 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:33:42.697666 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.697625 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:33:42.723183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.720269 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:33:42.775300 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.730655 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.802176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.776664 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:33:42.802176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.777520 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.802176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.777560 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.932523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.932475 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.977203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.967263 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.977203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:42.967324 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.157056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.156749 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.213503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.199848 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.266232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.264374 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.266232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.264451 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.336233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.314009 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.338320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.338281 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:43.387171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.386805 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.423178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.409814 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:33:43.471141 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.471107 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.471411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.471385 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.490175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.473264 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.491688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.491655 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:33:43.533236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.523516 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.538310 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.538277 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.679872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.679828 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.694376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.694258 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.694376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.694314 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.694774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.694678 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.694774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.694720 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.723180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.717719 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.723180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.718064 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 29 22:33:43.754771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.754728 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.756514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.756488 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:33:43.775060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.775011 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.899234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.880237 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.899234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.880298 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.942233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.942190 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.950590 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:43.950552 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.169233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.168882 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:33:44.338017 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.337982 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s"] Oct 29 22:33:44.341618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.341587 5073 cpu_manager.go:414] "containerMap: removing container" podUID="06266413-4748-40d7-b64f-68b301ddbec4" containerName="revoke-keys" Oct 29 22:33:44.341789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.341774 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06266413-4748-40d7-b64f-68b301ddbec4" containerName="revoke-keys" Oct 29 22:33:44.342210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.342189 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="06266413-4748-40d7-b64f-68b301ddbec4" containerName="revoke-keys" Oct 29 22:33:44.348425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.348392 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:33:44.348886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.348861 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:33:44.361407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.361381 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.381082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.379373 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:33:44.381082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.380033 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:44.391341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.391279 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.425074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.425006 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s"] Oct 29 22:33:44.495313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.495276 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.514924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.513436 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.514924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.513565 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.514924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.513681 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpfn\" (UniqueName: \"kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.545190 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.533846 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:33:44.545190 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.534178 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:33:44.548082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.546947 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:33:44.628428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.628392 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.638372 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.638335 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.641663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.641635 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpfn\" (UniqueName: \"kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.673478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.644017 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.717173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.665045 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.717173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.673818 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpfn\" (UniqueName: \"kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.717173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.698457 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f47dde-tpz5s\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:44.721168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.718697 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.759035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.758261 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:33:44.777184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.763234 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.781181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.777526 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.781181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:44.777980 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:33:45.055619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.055528 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.086689 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.086656 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:33:45.108198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.106649 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:33:45.108198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.106729 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:33:45.128832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.108455 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:45.128832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.120052 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:33:45.177669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.177624 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.198869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.190519 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:33:45.202629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.202597 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:33:45.211539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.211504 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.226471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.226446 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:33:45.239420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.239395 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:33:45.239549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.239461 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:33:45.240619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.240594 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:33:45.245424 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.245404 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:33:45.270181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.260108 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:33:45.526103 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:33:45.526078 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76efb132_ed2a_40d2_aab2_cf807aba073d.slice/crio-f03bc4e39bca40518f4060e00cc174ef7ea5e4b205b7e04b8323be7c0f95d78f WatchSource:0}: Error finding container f03bc4e39bca40518f4060e00cc174ef7ea5e4b205b7e04b8323be7c0f95d78f: Status 404 returned error can't find the container with id f03bc4e39bca40518f4060e00cc174ef7ea5e4b205b7e04b8323be7c0f95d78f Oct 29 22:33:45.526794 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:45.526766 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s"] Oct 29 22:33:46.104850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:46.104809 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" event={"ID":"76efb132-ed2a-40d2-aab2-cf807aba073d","Type":"ContainerStarted","Data":"1f22baa8109f5e62e2ef6d6a13733aa3e337078ee2bb944546107329a17af636"} Oct 29 22:33:46.105378 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:46.104899 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" event={"ID":"76efb132-ed2a-40d2-aab2-cf807aba073d","Type":"ContainerStarted","Data":"f03bc4e39bca40518f4060e00cc174ef7ea5e4b205b7e04b8323be7c0f95d78f"} Oct 29 22:33:46.147279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:46.147230 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" podStartSLOduration=4.147217316 podStartE2EDuration="4.147217316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:46.144678642 +0000 UTC m=+1141.460072724" watchObservedRunningTime="2026-10-29 22:33:46.147217316 +0000 UTC m=+1141.462611430" Oct 29 22:33:47.112632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:47.112597 5073 generic.go:356] "Generic (PLEG): container finished" podID="76efb132-ed2a-40d2-aab2-cf807aba073d" containerID="1f22baa8109f5e62e2ef6d6a13733aa3e337078ee2bb944546107329a17af636" exitCode=0 Oct 29 22:33:47.113098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:47.112676 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" event={"ID":"76efb132-ed2a-40d2-aab2-cf807aba073d","Type":"ContainerDied","Data":"1f22baa8109f5e62e2ef6d6a13733aa3e337078ee2bb944546107329a17af636"} Oct 29 22:33:48.307337 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.307312 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:48.394832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.394742 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn\" (UniqueName: \"kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn\") pod \"76efb132-ed2a-40d2-aab2-cf807aba073d\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " Oct 29 22:33:48.394992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.394934 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca\") pod \"76efb132-ed2a-40d2-aab2-cf807aba073d\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " Oct 29 22:33:48.395072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.395025 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca\") pod \"76efb132-ed2a-40d2-aab2-cf807aba073d\" (UID: \"76efb132-ed2a-40d2-aab2-cf807aba073d\") " Oct 29 22:33:48.395486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.395357 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca" pod "76efb132-ed2a-40d2-aab2-cf807aba073d" (UID: "76efb132-ed2a-40d2-aab2-cf807aba073d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:48.395787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.395762 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:48.397745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.397719 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn" pod "76efb132-ed2a-40d2-aab2-cf807aba073d" (UID: "76efb132-ed2a-40d2-aab2-cf807aba073d"). InnerVolumeSpecName "kube-api-access-srpfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:48.397850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.397811 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca" pod "76efb132-ed2a-40d2-aab2-cf807aba073d" (UID: "76efb132-ed2a-40d2-aab2-cf807aba073d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:33:48.498132 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.498101 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76efb132-ed2a-40d2-aab2-cf807aba073d-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:48.498267 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:48.498137 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srpfn\" (UniqueName: \"kubernetes.io/projected/76efb132-ed2a-40d2-aab2-cf807aba073d-kube-api-access-srpfn\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:49.131363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.131282 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" Oct 29 22:33:49.131363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.131317 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s" event={"ID":"76efb132-ed2a-40d2-aab2-cf807aba073d","Type":"ContainerDied","Data":"f03bc4e39bca40518f4060e00cc174ef7ea5e4b205b7e04b8323be7c0f95d78f"} Oct 29 22:33:49.131538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.131374 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f03bc4e39bca40518f4060e00cc174ef7ea5e4b205b7e04b8323be7c0f95d78f" Oct 29 22:33:49.943314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.943284 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9023-d8c18c"] Oct 29 22:33:49.946415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.946391 5073 cpu_manager.go:414] "containerMap: removing container" podUID="76efb132-ed2a-40d2-aab2-cf807aba073d" containerName="revoke-keys" Oct 29 22:33:49.946571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.946557 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76efb132-ed2a-40d2-aab2-cf807aba073d" containerName="revoke-keys" Oct 29 22:33:49.946867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.946850 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="76efb132-ed2a-40d2-aab2-cf807aba073d" containerName="revoke-keys" Oct 29 22:33:49.951772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.951739 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:49.968558 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.968529 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c5k5w\"" Oct 29 22:33:49.969645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:49.969623 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-d8c18c"] Oct 29 22:33:50.027097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:50.027055 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999rh\" (UniqueName: \"kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh\") pod \"test-curl-9023-d8c18c\" (UID: \"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9\") " pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:50.128808 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:50.128778 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999rh\" (UniqueName: \"kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh\") pod \"test-curl-9023-d8c18c\" (UID: \"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9\") " pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:50.149670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:50.149642 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999rh\" (UniqueName: \"kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh\") pod \"test-curl-9023-d8c18c\" (UID: \"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9\") " pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:50.283008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:50.282977 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:50.584441 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:33:50.584411 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14fe67e0_fc64_40fb_a1bf_ce3ee01170a9.slice/crio-de55f42ff9a6ca17d875ace7527f1a1b9218e90e9eeea84750f4d00338253d2b WatchSource:0}: Error finding container de55f42ff9a6ca17d875ace7527f1a1b9218e90e9eeea84750f4d00338253d2b: Status 404 returned error can't find the container with id de55f42ff9a6ca17d875ace7527f1a1b9218e90e9eeea84750f4d00338253d2b Oct 29 22:33:50.584585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:50.584488 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-d8c18c"] Oct 29 22:33:50.586456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:50.586436 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:33:51.148797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:51.148771 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-d8c18c" event={"ID":"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9","Type":"ContainerStarted","Data":"de55f42ff9a6ca17d875ace7527f1a1b9218e90e9eeea84750f4d00338253d2b"} Oct 29 22:33:52.158206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:52.158166 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-d8c18c" event={"ID":"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9","Type":"ContainerStarted","Data":"d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2"} Oct 29 22:33:52.200854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:52.200809 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9023-d8c18c" podStartSLOduration=2.716681381 podStartE2EDuration="3.200796275s" totalImagesPullingTime="484.114894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:52.198052079 +0000 UTC m=+1147.513446159" watchObservedRunningTime="2026-10-29 22:33:52.200796275 +0000 UTC m=+1147.516190355" Oct 29 22:33:53.044907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.044861 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9023-d8c18c"] Oct 29 22:33:53.051446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.051412 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9023-d8c18c"] Oct 29 22:33:53.204638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.204598 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9023-101e58"] Oct 29 22:33:53.205250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.205235 5073 cpu_manager.go:414] "containerMap: removing container" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" containerName="test-curl-9023-d8c18c" Oct 29 22:33:53.205312 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.205251 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" containerName="test-curl-9023-d8c18c" Oct 29 22:33:53.205384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.205376 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" containerName="test-curl-9023-d8c18c" Oct 29 22:33:53.209368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.209349 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:53.231530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.231497 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-101e58"] Oct 29 22:33:53.382216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.382119 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfrr\" (UniqueName: \"kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr\") pod \"test-curl-9023-101e58\" (UID: \"85c1fe0d-662e-41b2-be8a-1f18391b21d9\") " pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:53.483658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.483626 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfrr\" (UniqueName: \"kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr\") pod \"test-curl-9023-101e58\" (UID: \"85c1fe0d-662e-41b2-be8a-1f18391b21d9\") " pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:53.504830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.504800 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfrr\" (UniqueName: \"kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr\") pod \"test-curl-9023-101e58\" (UID: \"85c1fe0d-662e-41b2-be8a-1f18391b21d9\") " pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:53.528620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.528590 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:53.840383 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:53.840354 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-101e58"] Oct 29 22:33:53.841307 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:33:53.841270 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85c1fe0d_662e_41b2_be8a_1f18391b21d9.slice/crio-1541461ffb50fda093d6e22325ecaac11cd0f39b6cf19a080c53c1312d5294a8 WatchSource:0}: Error finding container 1541461ffb50fda093d6e22325ecaac11cd0f39b6cf19a080c53c1312d5294a8: Status 404 returned error can't find the container with id 1541461ffb50fda093d6e22325ecaac11cd0f39b6cf19a080c53c1312d5294a8 Oct 29 22:33:54.175366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:54.175281 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-101e58" event={"ID":"85c1fe0d-662e-41b2-be8a-1f18391b21d9","Type":"ContainerStarted","Data":"6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8"} Oct 29 22:33:54.175366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:54.175325 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-101e58" event={"ID":"85c1fe0d-662e-41b2-be8a-1f18391b21d9","Type":"ContainerStarted","Data":"1541461ffb50fda093d6e22325ecaac11cd0f39b6cf19a080c53c1312d5294a8"} Oct 29 22:33:54.175574 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:54.175545 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9023-d8c18c" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" containerName="test-curl-9023-d8c18c" containerID="cri-o://d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2" gracePeriod=2 Oct 29 22:33:54.216823 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:54.216776 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9023-101e58" podStartSLOduration=1.216763858 podStartE2EDuration="1.216763858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:54.215227681 +0000 UTC m=+1149.530621761" watchObservedRunningTime="2026-10-29 22:33:54.216763858 +0000 UTC m=+1149.532157937" Oct 29 22:33:54.230067 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:54.230041 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9023-d8c18c\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:54.851696 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:54.851662 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9023-d8c18c\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:55.096961 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:55.096930 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9023-101e58"] Oct 29 22:33:55.101622 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:55.101592 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9023-101e58"] Oct 29 22:33:56.191115 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:56.191080 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9023-101e58" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" containerName="test-curl-9023-101e58" containerID="cri-o://6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8" gracePeriod=2 Oct 29 22:33:56.205785 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:56.205758 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9023-101e58\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:56.448202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:56.448126 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:56.465355 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:56.465306 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9023-101e58\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:56.528235 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:56.528202 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh\" (UniqueName: \"kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh\") pod \"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9\" (UID: \"14fe67e0-fc64-40fb-a1bf-ce3ee01170a9\") " Oct 29 22:33:56.530941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:56.530904 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh" pod "14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" (UID: "14fe67e0-fc64-40fb-a1bf-ce3ee01170a9"). InnerVolumeSpecName "kube-api-access-999rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:56.650176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:56.633400 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999rh\" (UniqueName: \"kubernetes.io/projected/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9-kube-api-access-999rh\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:56.841515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:56.841481 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" path="/var/lib/kubelet/pods/14fe67e0-fc64-40fb-a1bf-ce3ee01170a9/volumes" Oct 29 22:33:57.202513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:57.202416 5073 generic.go:356] "Generic (PLEG): container finished" podID="14fe67e0-fc64-40fb-a1bf-ce3ee01170a9" containerID="d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2" exitCode=137 Oct 29 22:33:57.202513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:57.202484 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-d8c18c" Oct 29 22:33:57.202964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:57.202521 5073 scope.go:122] "RemoveContainer" containerID="d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2" Oct 29 22:33:57.214278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:57.214260 5073 scope.go:122] "RemoveContainer" containerID="d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2" Oct 29 22:33:57.214607 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:57.214589 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2\": container with ID starting with d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2 not found: ID does not exist" containerID="d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2" Oct 29 22:33:57.214682 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:57.214620 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2"} err="failed to get container status \"d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2\": rpc error: code = NotFound desc = could not find container \"d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2\": container with ID starting with d39797af0252e6f786171959851305a6ff27feaac498a8109fb5cb2fc2fb01c2 not found: ID does not exist" Oct 29 22:33:58.460390 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:58.460365 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:58.564409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:58.564382 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr\" (UniqueName: \"kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr\") pod \"85c1fe0d-662e-41b2-be8a-1f18391b21d9\" (UID: \"85c1fe0d-662e-41b2-be8a-1f18391b21d9\") " Oct 29 22:33:58.566908 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:58.566878 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr" pod "85c1fe0d-662e-41b2-be8a-1f18391b21d9" (UID: "85c1fe0d-662e-41b2-be8a-1f18391b21d9"). InnerVolumeSpecName "kube-api-access-qjfrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:58.666210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:58.666188 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjfrr\" (UniqueName: \"kubernetes.io/projected/85c1fe0d-662e-41b2-be8a-1f18391b21d9-kube-api-access-qjfrr\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:33:58.845424 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:58.845345 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" path="/var/lib/kubelet/pods/85c1fe0d-662e-41b2-be8a-1f18391b21d9/volumes" Oct 29 22:33:59.232870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:59.232818 5073 generic.go:356] "Generic (PLEG): container finished" podID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" containerID="6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8" exitCode=137 Oct 29 22:33:59.232870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:59.232871 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-101e58" Oct 29 22:33:59.233125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:59.232966 5073 scope.go:122] "RemoveContainer" containerID="6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8" Oct 29 22:33:59.250225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:59.250198 5073 scope.go:122] "RemoveContainer" containerID="6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8" Oct 29 22:33:59.250534 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:33:59.250515 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8\": container with ID starting with 6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8 not found: ID does not exist" containerID="6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8" Oct 29 22:33:59.250591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:33:59.250544 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8"} err="failed to get container status \"6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8\": rpc error: code = NotFound desc = could not find container \"6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8\": container with ID starting with 6d04af8aa09d5aefa802d72d89c4d09bdb6082d501ddfd48562d2269ac1e78e8 not found: ID does not exist" Oct 29 22:34:01.779809 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.779779 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl"] Oct 29 22:34:01.780445 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.780430 5073 cpu_manager.go:414] "containerMap: removing container" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" containerName="test-curl-9023-101e58" Oct 29 22:34:01.780508 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.780447 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" containerName="test-curl-9023-101e58" Oct 29 22:34:01.780572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.780563 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c1fe0d-662e-41b2-be8a-1f18391b21d9" containerName="test-curl-9023-101e58" Oct 29 22:34:01.784070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.784053 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:01.806486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.806459 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl"] Oct 29 22:34:01.877432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.877400 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:34:01.877603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.877580 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:34:01.910598 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.910559 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:01.910778 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.910744 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:01.910953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:01.910930 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczv7\" (UniqueName: \"kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.013297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.013264 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.013453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.013391 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.013551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.013534 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczv7\" (UniqueName: \"kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.013934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.013908 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.016279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.016254 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.036264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.036205 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczv7\" (UniqueName: \"kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7\") pod \"maas-api-revoke-sub-365e54a5-99qfl\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.098251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.098222 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:02.422430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:02.422397 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl"] Oct 29 22:34:02.422627 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:34:02.422605 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6843d41f_752c_40c6_95e5_047992996920.slice/crio-fdcec64e8fb0f3381c5d6a655ba272bf852a5eeccd0a7c731f4c0a57949c39ba WatchSource:0}: Error finding container fdcec64e8fb0f3381c5d6a655ba272bf852a5eeccd0a7c731f4c0a57949c39ba: Status 404 returned error can't find the container with id fdcec64e8fb0f3381c5d6a655ba272bf852a5eeccd0a7c731f4c0a57949c39ba Oct 29 22:34:03.306514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:03.306472 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" event={"ID":"6843d41f-752c-40c6-95e5-047992996920","Type":"ContainerStarted","Data":"d15408db911e31a6bb1c4180e2a0890e3d06c52b1d21330fd5170ae570e637ac"} Oct 29 22:34:03.306905 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:03.306540 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" event={"ID":"6843d41f-752c-40c6-95e5-047992996920","Type":"ContainerStarted","Data":"fdcec64e8fb0f3381c5d6a655ba272bf852a5eeccd0a7c731f4c0a57949c39ba"} Oct 29 22:34:03.351951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:03.351908 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" podStartSLOduration=2.351895631 podStartE2EDuration="2.351895631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:03.349656391 +0000 UTC m=+1158.665050495" watchObservedRunningTime="2026-10-29 22:34:03.351895631 +0000 UTC m=+1158.667289710" Oct 29 22:34:04.319058 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:04.319019 5073 generic.go:356] "Generic (PLEG): container finished" podID="6843d41f-752c-40c6-95e5-047992996920" containerID="d15408db911e31a6bb1c4180e2a0890e3d06c52b1d21330fd5170ae570e637ac" exitCode=0 Oct 29 22:34:04.319542 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:04.319097 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" event={"ID":"6843d41f-752c-40c6-95e5-047992996920","Type":"ContainerDied","Data":"d15408db911e31a6bb1c4180e2a0890e3d06c52b1d21330fd5170ae570e637ac"} Oct 29 22:34:05.508896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.508864 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:05.601594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.601499 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7\" (UniqueName: \"kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7\") pod \"6843d41f-752c-40c6-95e5-047992996920\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " Oct 29 22:34:05.601767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.601629 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca\") pod \"6843d41f-752c-40c6-95e5-047992996920\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " Oct 29 22:34:05.601890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.601869 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca\") pod \"6843d41f-752c-40c6-95e5-047992996920\" (UID: \"6843d41f-752c-40c6-95e5-047992996920\") " Oct 29 22:34:05.602121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.602092 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca" pod "6843d41f-752c-40c6-95e5-047992996920" (UID: "6843d41f-752c-40c6-95e5-047992996920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:05.603010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.602985 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843d41f-752c-40c6-95e5-047992996920-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:34:05.608225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.604448 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7" pod "6843d41f-752c-40c6-95e5-047992996920" (UID: "6843d41f-752c-40c6-95e5-047992996920"). InnerVolumeSpecName "kube-api-access-rczv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:05.608225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.606413 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca" pod "6843d41f-752c-40c6-95e5-047992996920" (UID: "6843d41f-752c-40c6-95e5-047992996920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:05.704448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.704417 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczv7\" (UniqueName: \"kubernetes.io/projected/6843d41f-752c-40c6-95e5-047992996920-kube-api-access-rczv7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:34:05.704448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:05.704445 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843d41f-752c-40c6-95e5-047992996920-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:34:06.285945 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:06.285906 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:34:06.335462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:06.335440 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" Oct 29 22:34:06.335618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:06.335474 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl" event={"ID":"6843d41f-752c-40c6-95e5-047992996920","Type":"ContainerDied","Data":"fdcec64e8fb0f3381c5d6a655ba272bf852a5eeccd0a7c731f4c0a57949c39ba"} Oct 29 22:34:06.335618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:06.335519 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdcec64e8fb0f3381c5d6a655ba272bf852a5eeccd0a7c731f4c0a57949c39ba" Oct 29 22:34:08.772977 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:08.772926 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:08.825303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:08.825255 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:08.839313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:08.839272 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:08.960002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:08.959953 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.193231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.185486 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.193231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.185932 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:34:09.368211 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.361095 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.368211 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.361500 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:34:09.405437 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.404170 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.461223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.454398 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.512172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.509558 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.547172 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:34:09.546817 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:34:09.564184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.549786 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.572207 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.569261 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.656339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.651236 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:09.856237 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.852930 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:34:09.885177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.880516 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:34:09.949175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.908117 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:34:09.949175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:09.908422 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:34:10.217247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.190836 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.229182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.192251 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.258057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.225751 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:34:10.258057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.239587 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:34:10.283176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.264235 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.363487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.363444 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.365335 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.365304 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:34:10.366879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.366858 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:34:10.367064 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.367050 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:34:10.408317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.408272 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.409727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.409704 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:34:10.550073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.549978 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.555584 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.555562 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:34:10.585177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.577354 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:34:10.585177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.577675 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:34:10.743789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.743759 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 22:34:10.923229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.912293 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.939035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:10.938986 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.127394 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.114836 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.204233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.163421 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 29 22:34:11.238179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.233286 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:11.368480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.368438 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.369051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.369023 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:34:11.413910 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.413869 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:34:11.660179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.604527 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.708176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.690407 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 29 22:34:11.708176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.691049 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:34:11.776610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.776557 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.943233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.923453 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:34:11.960284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.960242 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.977175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:11.969541 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.013020 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.012987 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:34:12.348174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.326860 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:12.615262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.614206 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 29 22:34:12.739178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.719077 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.739178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.719170 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.793186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.779908 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:34:12.793186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.780011 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.793186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.780055 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 29 22:34:12.802570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.795387 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.802570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.795718 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:34:12.802570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.795798 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:34:12.896366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.795651 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:34:12.896366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.875337 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:34:12.946824 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.946771 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.961081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.961034 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.969176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.962519 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.969414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:12.969383 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.081198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.060437 5073 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-h2w6v container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" start-of-body= Oct 29 22:34:13.081641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.081604 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2w6v" podUID="2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" Oct 29 22:34:13.081882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.081848 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:13.122659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.122625 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:13.122928 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.122888 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.139266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.139107 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 29 22:34:13.236777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.236747 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:34:13.307386 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.307345 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:34:13.397398 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.397220 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:34:13.397763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.397747 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:34:13.397928 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.397917 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:34:13.398032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.398024 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:34:13.414517 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.409970 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 22:34:13.423208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.422232 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:13.452171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.433398 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:34:13.452171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.440328 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:34:13.500228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.454091 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:34:13.500228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.454960 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:34:13.500228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.464653 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:34:13.500228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.483474 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:34:13.508492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.508459 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:34:13.543191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.543138 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:34:13.544503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:13.544483 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:34:18.428168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.425443 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw"] Oct 29 22:34:18.428168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.426553 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6843d41f-752c-40c6-95e5-047992996920" containerName="revoke-keys" Oct 29 22:34:18.428168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.426573 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6843d41f-752c-40c6-95e5-047992996920" containerName="revoke-keys" Oct 29 22:34:18.428168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.426951 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6843d41f-752c-40c6-95e5-047992996920" containerName="revoke-keys" Oct 29 22:34:18.434178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.433424 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.442488 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.442470 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw"] Oct 29 22:34:18.455769 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.455684 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f922c6e9-openshift-default-dockercfg-p5l5s\"" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.520894 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.520965 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521010 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521048 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521078 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521115 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521248 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521280 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmx5h\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.531224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.521316 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624102 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmx5h\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624187 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624346 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624404 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624453 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624501 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624536 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624580 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.624733 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.624722 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.649139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.639934 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.649139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.640202 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.649139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.640416 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.649139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.641017 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.649139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.641261 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.669221 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.667795 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.669895 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.669831 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.672635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.672608 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.676170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.674191 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmx5h\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h\") pod \"e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:18.776762 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:18.776727 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:19.142566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.139302 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw"] Oct 29 22:34:19.146636 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.146591 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:34:19.146847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.146821 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:34:19.146997 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.146977 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:34:19.374086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.374046 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" event={"ID":"3b05e417-7fd1-4965-af08-cd41eddec6f5","Type":"ContainerStarted","Data":"9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9"} Oct 29 22:34:19.374266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.374102 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" event={"ID":"3b05e417-7fd1-4965-af08-cd41eddec6f5","Type":"ContainerStarted","Data":"6bc11e8ccd98de77120c1d725b8f0246dfabd7b622c1755dd6764a7a3fa0c728"} Oct 29 22:34:19.423046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.422090 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podStartSLOduration=1.422073626 podStartE2EDuration="1.422073626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:19.419491237 +0000 UTC m=+1174.734885317" watchObservedRunningTime="2026-10-29 22:34:19.422073626 +0000 UTC m=+1174.737467708" Oct 29 22:34:19.779223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:19.779195 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:20.784643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:20.784611 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:21.393407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:21.393372 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:21.394100 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:21.394075 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:34:22.881656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:22.881624 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg"] Oct 29 22:34:22.886314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:22.886294 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:22.907275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:22.907249 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg"] Oct 29 22:34:22.909904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:22.909879 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1309d608-serving-cert\"" Oct 29 22:34:23.029667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.029635 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.029877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.029715 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.030027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.030010 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.030109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.030077 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4zd\" (UniqueName: \"kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.135653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.135556 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.135936 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.135904 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4zd\" (UniqueName: \"kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.136075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.136043 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.136185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.136115 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.141168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.140255 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.148222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.148128 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.153385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.153357 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.167524 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.166501 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4zd\" (UniqueName: \"kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd\") pod \"maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.207274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.207241 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:23.531099 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:34:23.531071 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e4be6f_6f37_4935_ae3c_06a285f48b29.slice/crio-219c0857171f9e3ecc7eca981ba16b0125a7405f55d192dec1bc5b2a3d139766 WatchSource:0}: Error finding container 219c0857171f9e3ecc7eca981ba16b0125a7405f55d192dec1bc5b2a3d139766: Status 404 returned error can't find the container with id 219c0857171f9e3ecc7eca981ba16b0125a7405f55d192dec1bc5b2a3d139766 Oct 29 22:34:23.533887 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:23.533863 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg"] Oct 29 22:34:24.419363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:24.419253 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" event={"ID":"06e4be6f-6f37-4935-ae3c-06a285f48b29","Type":"ContainerStarted","Data":"70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c"} Oct 29 22:34:24.419363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:24.419346 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" event={"ID":"06e4be6f-6f37-4935-ae3c-06a285f48b29","Type":"ContainerStarted","Data":"219c0857171f9e3ecc7eca981ba16b0125a7405f55d192dec1bc5b2a3d139766"} Oct 29 22:34:24.420510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:24.420474 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:24.462421 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:24.462380 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" podStartSLOduration=1.998571706 podStartE2EDuration="2.46236773s" totalImagesPullingTime="463.796024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:24.461483589 +0000 UTC m=+1179.776877668" watchObservedRunningTime="2026-10-29 22:34:24.46236773 +0000 UTC m=+1179.777761810" Oct 29 22:34:31.443424 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:31.443392 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:34:33.976303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:33.976245 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.382824 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.381480 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.395379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.395034 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.446402 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.445241 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:34:34.488300 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.487240 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 29 22:34:34.551562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.551503 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.608241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.608196 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.775177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.774230 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.878398 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.878355 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.940172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.937754 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 29 22:34:34.940172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:34.938799 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.113842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.113748 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.286549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.286499 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.344519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.286808 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 29 22:34:35.401222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.396642 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.426177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.404227 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:34:35.545177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.541679 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.545177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.543412 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:34:35.563521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.563483 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:35.565641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.565604 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:34:35.565930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.565912 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:34:35.578846 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.567788 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.578846 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.569110 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:34:35.634282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.634237 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.775101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.775023 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:34:35.789965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.789934 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:34:35.958901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.947665 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.958901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.947902 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:34:35.958901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.948058 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.958901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.951794 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:34:35.958901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.951852 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:34:35.958901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:35.951891 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:34:36.248173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.242539 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:34:36.329180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.311624 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:36.353174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.350647 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:34:36.472258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.472216 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.576925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.576833 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:36.577873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.577835 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:34:36.580060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.580031 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.768757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.768711 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.786679 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.786639 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 29 22:34:36.948340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.948217 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:34:36.949401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.949365 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:34:36.991785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:36.991298 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.079460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.079414 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:34:37.247010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.246961 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.312301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.312239 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.473053 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.473006 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.583317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.583219 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.585395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.583551 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:37.772278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.772233 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:34:37.958234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:37.955423 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.018373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.005289 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:38.018373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.005372 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.018373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.005522 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.652693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.584473 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.652693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.585167 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:34:38.652693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.649258 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7"] Oct 29 22:34:38.763969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.763930 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:34:38.764371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.764352 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:34:38.764577 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.764562 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:34:38.786599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.786575 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:38.857184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.840422 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3593b8ba-serving-cert\"" Oct 29 22:34:38.869185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.852463 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.869185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.852693 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:38.869185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.852907 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.869185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.866408 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.911379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.907635 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:34:38.911379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.907846 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.995066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:38.995020 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:34:39.122177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.112362 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:39.172223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.167867 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:39.263218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.263142 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:39.265011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.264984 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 29 22:34:39.493983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.493950 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7"] Oct 29 22:34:39.494416 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.494395 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:34:39.494585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.494571 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 22:34:39.494744 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.494730 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:34:39.498692 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.498674 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:34:39.500442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.500407 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 29 22:34:39.526370 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.526346 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 22:34:39.610094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.610053 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.610311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.610173 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.610311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.610226 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.610311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.610252 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbl2f\" (UniqueName: \"kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.635898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.635868 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 22:34:39.649646 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:34:39.649613 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:34:39.714405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.714352 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.721382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.718561 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.721382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.718687 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.721382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.718724 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbl2f\" (UniqueName: \"kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.721382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.720734 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.724671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.724640 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.739297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.739266 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.783904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.783837 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbl2f\" (UniqueName: \"kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f\") pod \"maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:39.902478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:39.902447 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:40.228654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.228624 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7"] Oct 29 22:34:40.229484 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:34:40.229457 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3c9b80_b85a_463d_b974_f6f720e37067.slice/crio-efb208e4abeac11fa67f81c44662237a3decc12271777c2d0c78f5f3d85a7687 WatchSource:0}: Error finding container efb208e4abeac11fa67f81c44662237a3decc12271777c2d0c78f5f3d85a7687: Status 404 returned error can't find the container with id efb208e4abeac11fa67f81c44662237a3decc12271777c2d0c78f5f3d85a7687 Oct 29 22:34:40.334791 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.334758 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" event={"ID":"4b3c9b80-b85a-463d-b974-f6f720e37067","Type":"ContainerStarted","Data":"efb208e4abeac11fa67f81c44662237a3decc12271777c2d0c78f5f3d85a7687"} Oct 29 22:34:40.960271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.960241 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn"] Oct 29 22:34:40.965447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.965425 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:40.984124 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.984091 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn"] Oct 29 22:34:40.992564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.992526 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:34:40.992564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.992527 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:34:40.992860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.992837 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:34:40.992980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:40.992929 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cvlmz\"" Oct 29 22:34:41.039236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039201 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.039411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039322 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.039411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039380 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.039563 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039545 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.039626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039595 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcwl\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.039697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039681 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.039755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.039726 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.141591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141557 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.141821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141606 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcwl\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.141821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141682 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.141821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141711 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.141821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141800 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.142054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141878 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.142054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.141927 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.143319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.142813 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.144764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.144737 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.145334 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.145308 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.145559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.145535 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.145640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.145610 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.164422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.164387 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcwl\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.166686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.166654 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-st7pn\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.280207 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.280105 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:41.349299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.349261 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" event={"ID":"4b3c9b80-b85a-463d-b974-f6f720e37067","Type":"ContainerStarted","Data":"d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e"} Oct 29 22:34:41.351528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.351456 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:34:41.395142 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.395083 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" podStartSLOduration=4.006534172 podStartE2EDuration="4.395062602s" totalImagesPullingTime="388.52843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:41.393614109 +0000 UTC m=+1196.709008188" watchObservedRunningTime="2026-10-29 22:34:41.395062602 +0000 UTC m=+1196.710456684" Oct 29 22:34:41.611716 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:34:41.611682 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3546bd90_f494_41cb_97a5_f53282656cd8.slice/crio-adf2f99136cc86edeecaba6c49d8e69ae153603c220b78a56d2fc123f50fa9d7 WatchSource:0}: Error finding container adf2f99136cc86edeecaba6c49d8e69ae153603c220b78a56d2fc123f50fa9d7: Status 404 returned error can't find the container with id adf2f99136cc86edeecaba6c49d8e69ae153603c220b78a56d2fc123f50fa9d7 Oct 29 22:34:41.611839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:41.611722 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn"] Oct 29 22:34:42.360957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:42.360903 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" event={"ID":"3546bd90-f494-41cb-97a5-f53282656cd8","Type":"ContainerStarted","Data":"adf2f99136cc86edeecaba6c49d8e69ae153603c220b78a56d2fc123f50fa9d7"} Oct 29 22:34:44.534410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:44.534378 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:34:44.534749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:44.534442 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:34:44.995298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:44.995267 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:34:44.998140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:44.998118 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:34:45.000477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.000453 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:34:45.003186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.003168 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:34:45.401749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.401687 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" event={"ID":"3546bd90-f494-41cb-97a5-f53282656cd8","Type":"ContainerStarted","Data":"12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f"} Oct 29 22:34:45.401986 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.401780 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:45.404201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.404132 5073 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-st7pn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:34:45.404298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.404204 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" podUID="3546bd90-f494-41cb-97a5-f53282656cd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:34:45.479820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:45.479769 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" podStartSLOduration=2.560671911 podStartE2EDuration="5.479755207s" totalImagesPullingTime="2.919083296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:45.461963856 +0000 UTC m=+1200.777357937" watchObservedRunningTime="2026-10-29 22:34:45.479755207 +0000 UTC m=+1200.795149286" Oct 29 22:34:46.409318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:46.409285 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:34:48.375401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:34:48.375372 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:35:08.051231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:08.050296 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h"] Oct 29 22:35:08.062919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:08.062885 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba896a18-2r24h"] Oct 29 22:35:08.842750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:08.842716 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3344d5cb-ce63-441b-a72f-a34dc68b0e01" path="/var/lib/kubelet/pods/3344d5cb-ce63-441b-a72f-a34dc68b0e01/volumes" Oct 29 22:35:08.888768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:08.888740 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:35:09.257925 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:35:09.257891 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:35:12.055929 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.040604 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs"] Oct 29 22:35:12.081179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.070652 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83725a30-cn8zs"] Oct 29 22:35:12.764191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.762579 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:35:12.833177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.823264 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.876524 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.876240 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.885177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.876803 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.913177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.911305 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12bd78b-7c5f-4b60-987e-97688149a7e9" path="/var/lib/kubelet/pods/c12bd78b-7c5f-4b60-987e-97688149a7e9/volumes" Oct 29 22:35:12.969175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.968236 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.969175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.968812 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:35:13.000176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:12.999240 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.356486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.341449 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.508176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.504694 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:35:13.808179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.807253 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.824272 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.824232 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.825030 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.825002 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:35:13.884525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.884482 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.917920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.917891 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:35:13.998386 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.998338 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 29 22:35:13.998922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:13.998895 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:35:14.012814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.011279 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 29 22:35:14.346283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.346237 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:14.346893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.346746 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:35:14.698232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.677142 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 29 22:35:14.698232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.697796 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:35:14.811307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.811258 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:14.839173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.814216 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 29 22:35:14.851175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:14.839822 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.165084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.165040 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:35:15.166616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.166585 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 29 22:35:15.168009 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.020496 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.169405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.020791 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.192173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.020432 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.206194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.205934 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.277199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.266917 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.278193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.278136 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:35:15.371265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.315758 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.445850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.445752 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:35:15.571113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.571069 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.636674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.572445 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.674646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.673671 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:35:15.674646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.673738 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 22:35:15.674646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.673771 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 22:35:15.674646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.673799 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 22:35:15.674646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.673826 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 22:35:15.689449 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.689415 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 22:35:15.702143 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:15.702071 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 22:35:19.887116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:19.887083 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q"] Oct 29 22:35:19.900298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:19.900271 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:19.914424 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:19.914399 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q"] Oct 29 22:35:19.996552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:19.996518 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3593b8ba-gw-openshift-default-dockercfg-skwqf\"" Oct 29 22:35:20.065482 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.065443 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kfc\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.065661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.065560 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.065661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.065630 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.065838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.065809 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.065977 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.065959 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.066034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.066013 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.066199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.066131 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.066473 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.066433 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.066582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.066478 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.168637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168551 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.168637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168601 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.168874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168677 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kfc\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.168874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168727 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.168874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168803 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168870 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168930 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.168962 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.169005 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.169049 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.169218 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.169724 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.169699 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.170194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.170168 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.171261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.171235 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.173198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.173166 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.173384 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.173365 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.191367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.191123 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.191367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.191326 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kfc\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.224513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.224477 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:20.625344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.625310 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q"] Oct 29 22:35:20.626491 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:35:20.626455 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4683b0_7c16_4d81_b0b0_e585233f3a7e.slice/crio-000d0c63d999f1f953ad98058523fdd341997119168d3e6262d6f948b1b271c3 WatchSource:0}: Error finding container 000d0c63d999f1f953ad98058523fdd341997119168d3e6262d6f948b1b271c3: Status 404 returned error can't find the container with id 000d0c63d999f1f953ad98058523fdd341997119168d3e6262d6f948b1b271c3 Oct 29 22:35:20.630331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.630289 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:35:20.630477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.630374 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:35:20.630477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.630418 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:35:20.685178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.684189 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" event={"ID":"df4683b0-7c16-4d81-b0b0-e585233f3a7e","Type":"ContainerStarted","Data":"000d0c63d999f1f953ad98058523fdd341997119168d3e6262d6f948b1b271c3"} Oct 29 22:35:20.756852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.756816 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg"] Oct 29 22:35:20.757095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:20.757055 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" podUID="06e4be6f-6f37-4935-ae3c-06a285f48b29" containerName="maas-api" containerID="cri-o://70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c" gracePeriod=30 Oct 29 22:35:21.080607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.080584 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:35:21.273181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.259986 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs\") pod \"06e4be6f-6f37-4935-ae3c-06a285f48b29\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " Oct 29 22:35:21.283180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.273850 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls\") pod \"06e4be6f-6f37-4935-ae3c-06a285f48b29\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " Oct 29 22:35:21.290708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.283572 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd\" (UniqueName: \"kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd\") pod \"06e4be6f-6f37-4935-ae3c-06a285f48b29\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " Oct 29 22:35:21.291523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.291498 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca\") pod \"06e4be6f-6f37-4935-ae3c-06a285f48b29\" (UID: \"06e4be6f-6f37-4935-ae3c-06a285f48b29\") " Oct 29 22:35:21.292180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.292065 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca" pod "06e4be6f-6f37-4935-ae3c-06a285f48b29" (UID: "06e4be6f-6f37-4935-ae3c-06a285f48b29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:21.293300 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.293277 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e4be6f-6f37-4935-ae3c-06a285f48b29-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:21.312124 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.309933 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs" pod "06e4be6f-6f37-4935-ae3c-06a285f48b29" (UID: "06e4be6f-6f37-4935-ae3c-06a285f48b29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:21.335035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.334982 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls" pod "06e4be6f-6f37-4935-ae3c-06a285f48b29" (UID: "06e4be6f-6f37-4935-ae3c-06a285f48b29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:21.338132 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.337306 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd" pod "06e4be6f-6f37-4935-ae3c-06a285f48b29" (UID: "06e4be6f-6f37-4935-ae3c-06a285f48b29"). InnerVolumeSpecName "kube-api-access-vf4zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:21.395032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.394995 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:21.395295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.395269 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e4be6f-6f37-4935-ae3c-06a285f48b29-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:21.395295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.395295 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4zd\" (UniqueName: \"kubernetes.io/projected/06e4be6f-6f37-4935-ae3c-06a285f48b29-kube-api-access-vf4zd\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:21.713185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.710313 5073 generic.go:356] "Generic (PLEG): container finished" podID="06e4be6f-6f37-4935-ae3c-06a285f48b29" containerID="70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c" exitCode=0 Oct 29 22:35:21.713185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.710445 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" event={"ID":"06e4be6f-6f37-4935-ae3c-06a285f48b29","Type":"ContainerDied","Data":"70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c"} Oct 29 22:35:21.713185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.710495 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" event={"ID":"06e4be6f-6f37-4935-ae3c-06a285f48b29","Type":"ContainerDied","Data":"219c0857171f9e3ecc7eca981ba16b0125a7405f55d192dec1bc5b2a3d139766"} Oct 29 22:35:21.713185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.710523 5073 scope.go:122] "RemoveContainer" containerID="70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c" Oct 29 22:35:21.713185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.710674 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg" Oct 29 22:35:21.739591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.739541 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" event={"ID":"df4683b0-7c16-4d81-b0b0-e585233f3a7e","Type":"ContainerStarted","Data":"e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb"} Oct 29 22:35:21.784297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.784207 5073 scope.go:122] "RemoveContainer" containerID="70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c" Oct 29 22:35:21.808061 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:35:21.808017 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c\": container with ID starting with 70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c not found: ID does not exist" containerID="70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c" Oct 29 22:35:21.808344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.808309 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c"} err="failed to get container status \"70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c\": rpc error: code = NotFound desc = could not find container \"70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c\": container with ID starting with 70612980776fd9b2f1c1e14850a74a8188e4c23be93379ce27e8b60a71f0942c not found: ID does not exist" Oct 29 22:35:21.849670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.849579 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podStartSLOduration=2.849554946 podStartE2EDuration="2.849554946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:21.805192106 +0000 UTC m=+1237.120586189" watchObservedRunningTime="2026-10-29 22:35:21.849554946 +0000 UTC m=+1237.164949028" Oct 29 22:35:21.880089 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.880058 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg"] Oct 29 22:35:21.884419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:21.884393 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1309d608-78d8d5dfd5-lhscg"] Oct 29 22:35:22.235687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:22.233848 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:22.235687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:22.233993 5073 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 29 22:35:22.235687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:22.234050 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 29 22:35:22.454176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:22.440267 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:35:22.454176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:22.440343 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:22.864902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:22.864852 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e4be6f-6f37-4935-ae3c-06a285f48b29" path="/var/lib/kubelet/pods/06e4be6f-6f37-4935-ae3c-06a285f48b29/volumes" Oct 29 22:35:23.225751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:23.225716 5073 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 29 22:35:23.226016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:23.225987 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 29 22:35:24.226531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.225982 5073 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 29 22:35:24.226531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.226097 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 29 22:35:24.790404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.790325 5073 generic.go:356] "Generic (PLEG): container finished" podID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerID="9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9" exitCode=0 Oct 29 22:35:24.790404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.790362 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" event={"ID":"3b05e417-7fd1-4965-af08-cd41eddec6f5","Type":"ContainerDied","Data":"9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9"} Oct 29 22:35:24.790956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.790925 5073 scope.go:122] "RemoveContainer" containerID="9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9" Oct 29 22:35:24.792629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.792597 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:35:24.792735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.792670 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:35:24.792735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:24.792704 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:35:25.234174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:25.226485 5073 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 29 22:35:25.234174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:25.226545 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 29 22:35:25.802523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:25.802490 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" event={"ID":"3b05e417-7fd1-4965-af08-cd41eddec6f5","Type":"ContainerStarted","Data":"6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024"} Oct 29 22:35:26.225653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:26.225619 5073 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 29 22:35:26.225992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:26.225954 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 29 22:35:26.968175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:26.904166 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:35:26.968175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:26.904232 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:35:27.102139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:27.101554 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:35:27.233229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:27.228606 5073 patch_prober.go:29] interesting pod/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 29 22:35:27.233229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:27.228683 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 29 22:35:27.779307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:27.779273 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:35:27.779471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:27.779325 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:35:28.244172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:28.243446 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:28.245291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:28.245265 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:28.253756 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:28.253730 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:28.778685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:28.778649 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:35:28.778871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:28.778702 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:35:29.778666 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:29.778625 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:35:29.779187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:29.778691 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:35:30.779054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:30.779019 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:35:30.779475 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:30.779071 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:35:31.871178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:31.867029 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:35:31.871178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:31.868646 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:35:31.872428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:31.872395 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:35:34.136330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.136293 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7"] Oct 29 22:35:34.136827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.136700 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" podUID="4b3c9b80-b85a-463d-b974-f6f720e37067" containerName="maas-api" containerID="cri-o://d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e" gracePeriod=30 Oct 29 22:35:34.639478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.639448 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:35:34.754565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.754533 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls\") pod \"4b3c9b80-b85a-463d-b974-f6f720e37067\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " Oct 29 22:35:34.754764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.754589 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f\" (UniqueName: \"kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f\") pod \"4b3c9b80-b85a-463d-b974-f6f720e37067\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " Oct 29 22:35:34.754764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.754742 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca\") pod \"4b3c9b80-b85a-463d-b974-f6f720e37067\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " Oct 29 22:35:34.754904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.754879 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs\") pod \"4b3c9b80-b85a-463d-b974-f6f720e37067\" (UID: \"4b3c9b80-b85a-463d-b974-f6f720e37067\") " Oct 29 22:35:34.766456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.757630 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca" pod "4b3c9b80-b85a-463d-b974-f6f720e37067" (UID: "4b3c9b80-b85a-463d-b974-f6f720e37067"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:34.766456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.758681 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f" pod "4b3c9b80-b85a-463d-b974-f6f720e37067" (UID: "4b3c9b80-b85a-463d-b974-f6f720e37067"). InnerVolumeSpecName "kube-api-access-gbl2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:34.766456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.758762 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3c9b80-b85a-463d-b974-f6f720e37067-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:34.766456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.766231 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs" pod "4b3c9b80-b85a-463d-b974-f6f720e37067" (UID: "4b3c9b80-b85a-463d-b974-f6f720e37067"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:34.768260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.768229 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls" pod "4b3c9b80-b85a-463d-b974-f6f720e37067" (UID: "4b3c9b80-b85a-463d-b974-f6f720e37067"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:34.859890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.859834 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:34.859890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.859859 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbl2f\" (UniqueName: \"kubernetes.io/projected/4b3c9b80-b85a-463d-b974-f6f720e37067-kube-api-access-gbl2f\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:34.859890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:34.859870 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b3c9b80-b85a-463d-b974-f6f720e37067-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:35.140090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.138294 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q"] Oct 29 22:35:35.140090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.138505 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" containerID="cri-o://e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb" gracePeriod=30 Oct 29 22:35:35.204724 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.204692 5073 generic.go:356] "Generic (PLEG): container finished" podID="4b3c9b80-b85a-463d-b974-f6f720e37067" containerID="d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e" exitCode=0 Oct 29 22:35:35.204901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.204843 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" event={"ID":"4b3c9b80-b85a-463d-b974-f6f720e37067","Type":"ContainerDied","Data":"d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e"} Oct 29 22:35:35.204901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.204884 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" event={"ID":"4b3c9b80-b85a-463d-b974-f6f720e37067","Type":"ContainerDied","Data":"efb208e4abeac11fa67f81c44662237a3decc12271777c2d0c78f5f3d85a7687"} Oct 29 22:35:35.204982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.204903 5073 scope.go:122] "RemoveContainer" containerID="d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e" Oct 29 22:35:35.205049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.205038 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7" Oct 29 22:35:35.230699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.230673 5073 scope.go:122] "RemoveContainer" containerID="d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e" Oct 29 22:35:35.231067 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:35:35.231043 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e\": container with ID starting with d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e not found: ID does not exist" containerID="d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e" Oct 29 22:35:35.231217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.231078 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e"} err="failed to get container status \"d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e\": rpc error: code = NotFound desc = could not find container \"d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e\": container with ID starting with d42745ef1cd0f3f05a7a82105628f74a707bae3c711f9816df70f5f5e3ace33e not found: ID does not exist" Oct 29 22:35:35.271460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.271433 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7"] Oct 29 22:35:35.275823 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:35.275786 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3593b8ba-db55b7746-59mk7"] Oct 29 22:35:36.840263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:36.840231 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3c9b80-b85a-463d-b974-f6f720e37067" path="/var/lib/kubelet/pods/4b3c9b80-b85a-463d-b974-f6f720e37067/volumes" Oct 29 22:35:39.257673 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:35:39.257633 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:35:40.438519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.438490 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:40.546546 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546512 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.546753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546565 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.546753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546626 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.546888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546755 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.546888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546809 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.546888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546861 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.547044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.546933 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.547102 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547085 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.547209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547130 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.547209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547178 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data\") pod \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\" (UID: \"df4683b0-7c16-4d81-b0b0-e585233f3a7e\") " Oct 29 22:35:40.547655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547122 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.547655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547589 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.547812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547653 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.547812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.547726 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:40.548559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.548459 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.548559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.548505 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.548559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.548535 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.548559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.548549 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.548559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.548564 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.555367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.553507 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:35:40.555367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.553547 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.555367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.553581 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "kube-api-access-49kfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:40.556561 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.556538 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token" pod "df4683b0-7c16-4d81-b0b0-e585233f3a7e" (UID: "df4683b0-7c16-4d81-b0b0-e585233f3a7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:40.649680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.649642 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.649680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.649671 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.649680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.649684 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kfc\" (UniqueName: \"kubernetes.io/projected/df4683b0-7c16-4d81-b0b0-e585233f3a7e-kube-api-access-49kfc\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.649922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:40.649698 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4683b0-7c16-4d81-b0b0-e585233f3a7e-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:35:41.250632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.250598 5073 generic.go:356] "Generic (PLEG): container finished" podID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerID="e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb" exitCode=0 Oct 29 22:35:41.250800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.250644 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" Oct 29 22:35:41.250800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.250674 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" event={"ID":"df4683b0-7c16-4d81-b0b0-e585233f3a7e","Type":"ContainerDied","Data":"e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb"} Oct 29 22:35:41.250800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.250731 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q" event={"ID":"df4683b0-7c16-4d81-b0b0-e585233f3a7e","Type":"ContainerDied","Data":"000d0c63d999f1f953ad98058523fdd341997119168d3e6262d6f948b1b271c3"} Oct 29 22:35:41.250800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.250753 5073 scope.go:122] "RemoveContainer" containerID="e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb" Oct 29 22:35:41.261366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.261179 5073 scope.go:122] "RemoveContainer" containerID="e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb" Oct 29 22:35:41.261486 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:35:41.261463 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb\": container with ID starting with e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb not found: ID does not exist" containerID="e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb" Oct 29 22:35:41.261531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.261495 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb"} err="failed to get container status \"e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb\": rpc error: code = NotFound desc = could not find container \"e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb\": container with ID starting with e6ef499fb893ec7fa7ecce7cda7d79d095f51f796dde15565ef9d7fa87ff67bb not found: ID does not exist" Oct 29 22:35:41.322686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.322661 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q"] Oct 29 22:35:41.334981 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:41.334958 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-2829q"] Oct 29 22:35:42.839612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:35:42.839581 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" path="/var/lib/kubelet/pods/df4683b0-7c16-4d81-b0b0-e585233f3a7e/volumes" Oct 29 22:36:09.255673 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:36:09.255636 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:36:18.698937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:18.698887 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:36:39.264497 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:36:39.264446 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:36:45.343328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:45.343287 5073 scope.go:122] "RemoveContainer" containerID="435aca19a9ab8e93b359afeec6717cb1c534ac09e63a67b198ec98848380ce95" Oct 29 22:36:45.357810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:45.357785 5073 scope.go:122] "RemoveContainer" containerID="74d736d8acb43a23a76a962ffd096d5b06c947f3bbe35427fe8cea6a5d1e48ed" Oct 29 22:36:55.675867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.675835 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt"] Oct 29 22:36:55.676763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676738 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4b3c9b80-b85a-463d-b974-f6f720e37067" containerName="maas-api" Oct 29 22:36:55.676763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676761 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3c9b80-b85a-463d-b974-f6f720e37067" containerName="maas-api" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676786 5073 cpu_manager.go:414] "containerMap: removing container" podUID="06e4be6f-6f37-4935-ae3c-06a285f48b29" containerName="maas-api" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676791 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e4be6f-6f37-4935-ae3c-06a285f48b29" containerName="maas-api" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676799 5073 cpu_manager.go:414] "containerMap: removing container" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676806 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676951 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3c9b80-b85a-463d-b974-f6f720e37067" containerName="maas-api" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676963 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e4be6f-6f37-4935-ae3c-06a285f48b29" containerName="maas-api" Oct 29 22:36:55.676970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.676970 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4683b0-7c16-4d81-b0b0-e585233f3a7e" containerName="istio-proxy" Oct 29 22:36:55.680662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.680646 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.700298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.700270 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt"] Oct 29 22:36:55.708016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.707980 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-afdbe8-serving-cert\"" Oct 29 22:36:55.762828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.762798 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.762992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.762835 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfjb\" (UniqueName: \"kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.762992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.762927 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.763088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.763065 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.864282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.864240 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.864474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.864286 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfjb\" (UniqueName: \"kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.864474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.864336 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.864596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.864500 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.865029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.865003 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.866839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.866810 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.866925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.866911 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.886105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.886069 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfjb\" (UniqueName: \"kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb\") pod \"maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:55.995572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:55.995517 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:56.314996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:56.314963 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt"] Oct 29 22:36:56.315755 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:36:56.315728 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b62beb_82c4_4577_bf93_cc54642f96f9.slice/crio-7e4bd409f443dfe60b608735914c29929a339172315cbfd26b743c58940943a8 WatchSource:0}: Error finding container 7e4bd409f443dfe60b608735914c29929a339172315cbfd26b743c58940943a8: Status 404 returned error can't find the container with id 7e4bd409f443dfe60b608735914c29929a339172315cbfd26b743c58940943a8 Oct 29 22:36:57.057002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:57.056918 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" event={"ID":"20b62beb-82c4-4577-bf93-cc54642f96f9","Type":"ContainerStarted","Data":"c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61"} Oct 29 22:36:57.057002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:57.056972 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" event={"ID":"20b62beb-82c4-4577-bf93-cc54642f96f9","Type":"ContainerStarted","Data":"7e4bd409f443dfe60b608735914c29929a339172315cbfd26b743c58940943a8"} Oct 29 22:36:57.057002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:57.056994 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:36:57.107486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:36:57.107419 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" podStartSLOduration=1.703043837 podStartE2EDuration="2.107403773s" totalImagesPullingTime="404.359936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:57.101965021 +0000 UTC m=+1332.417359101" watchObservedRunningTime="2026-10-29 22:36:57.107403773 +0000 UTC m=+1332.422797853" Oct 29 22:37:03.070900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:03.070872 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 22:37:09.261898 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:37:09.261860 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:37:14.165188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:14.165135 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw"] Oct 29 22:37:14.165680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:14.165410 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" containerID="cri-o://6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024" gracePeriod=30 Oct 29 22:37:16.871121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:16.871077 5073 patch_prober.go:29] interesting pod/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:37:16.871526 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:16.871199 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:37:19.484647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.484617 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:37:19.592191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592133 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592271 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592339 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592513 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592573 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592690 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592671 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592715 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592724 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:19.592868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592809 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.592993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.592873 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy\") pod \"3b05e417-7fd1-4965-af08-cd41eddec6f5\" (UID: \"3b05e417-7fd1-4965-af08-cd41eddec6f5\") " Oct 29 22:37:19.593059 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.593028 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:19.593242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.593215 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:19.593569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.593544 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:19.593841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.593764 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b05e417-7fd1-4965-af08-cd41eddec6f5-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.593974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.593941 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.593974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.593968 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.594202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.594136 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:19.595650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.595611 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "kube-api-access-jmx5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:19.595873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.595851 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:19.596039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.596017 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:19.596322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.596303 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo" pod "3b05e417-7fd1-4965-af08-cd41eddec6f5" (UID: "3b05e417-7fd1-4965-af08-cd41eddec6f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:37:19.695003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.694983 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.695003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.695004 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.695133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.695012 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmx5h\" (UniqueName: \"kubernetes.io/projected/3b05e417-7fd1-4965-af08-cd41eddec6f5-kube-api-access-jmx5h\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.695133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.695021 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b05e417-7fd1-4965-af08-cd41eddec6f5-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.695133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.695029 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:19.695133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:19.695037 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b05e417-7fd1-4965-af08-cd41eddec6f5-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:37:20.234444 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.234414 5073 generic.go:356] "Generic (PLEG): container finished" podID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerID="6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024" exitCode=0 Oct 29 22:37:20.234607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.234477 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" Oct 29 22:37:20.234607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.234499 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" event={"ID":"3b05e417-7fd1-4965-af08-cd41eddec6f5","Type":"ContainerDied","Data":"6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024"} Oct 29 22:37:20.234607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.234556 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw" event={"ID":"3b05e417-7fd1-4965-af08-cd41eddec6f5","Type":"ContainerDied","Data":"6bc11e8ccd98de77120c1d725b8f0246dfabd7b622c1755dd6764a7a3fa0c728"} Oct 29 22:37:20.234607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.234581 5073 scope.go:122] "RemoveContainer" containerID="6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024" Oct 29 22:37:20.245689 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.245671 5073 scope.go:122] "RemoveContainer" containerID="9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9" Oct 29 22:37:20.255073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.255054 5073 scope.go:122] "RemoveContainer" containerID="6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024" Oct 29 22:37:20.255339 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:37:20.255321 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024\": container with ID starting with 6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024 not found: ID does not exist" containerID="6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024" Oct 29 22:37:20.255407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.255348 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024"} err="failed to get container status \"6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024\": rpc error: code = NotFound desc = could not find container \"6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024\": container with ID starting with 6dbdf99e1af96bb7c9fa7a9d61ca940c2f1e58476691e4e902c5d2aa52a03024 not found: ID does not exist" Oct 29 22:37:20.255407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.255365 5073 scope.go:122] "RemoveContainer" containerID="9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9" Oct 29 22:37:20.255642 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:37:20.255628 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9\": container with ID starting with 9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9 not found: ID does not exist" containerID="9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9" Oct 29 22:37:20.255687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.255646 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9"} err="failed to get container status \"9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9\": rpc error: code = NotFound desc = could not find container \"9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9\": container with ID starting with 9583867fb9f86f74d65b37af42e7498b5f53a80fbad2ee52b5a71a078f2b27b9 not found: ID does not exist" Oct 29 22:37:20.304806 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.304779 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw"] Oct 29 22:37:20.318084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.318062 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f922c6e9-openshift-default-797b4f7c65-dxvhw"] Oct 29 22:37:20.839919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:20.839891 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" path="/var/lib/kubelet/pods/3b05e417-7fd1-4965-af08-cd41eddec6f5/volumes" Oct 29 22:37:21.048511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:21.048477 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:37:24.354840 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.354793 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x"] Oct 29 22:37:24.356453 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.356397 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" Oct 29 22:37:24.356641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.356626 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" Oct 29 22:37:24.356735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.356723 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" Oct 29 22:37:24.356825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.356812 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" Oct 29 22:37:24.357164 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.357134 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" Oct 29 22:37:24.357282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.357272 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b05e417-7fd1-4965-af08-cd41eddec6f5" containerName="istio-proxy" Oct 29 22:37:24.384734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.384705 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.387073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.387047 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x"] Oct 29 22:37:24.404188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.403938 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-c7b96a-openshift-default-dockercfg-8qk9p\"" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515000 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515106 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515137 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515260 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515286 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515309 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515350 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515506 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.516761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.515625 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srb2b\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617194 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617233 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617258 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617297 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617453 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617576 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srb2b\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617622 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617661 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617699 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617744 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.617906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.617771 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.618203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.618170 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.618355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.618181 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.618355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.618263 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.620814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.620793 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.621140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.621119 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.639992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.639952 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.640439 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.640416 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srb2b\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b\") pod \"e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:24.827266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:24.827232 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:25.184933 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:37:25.184903 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6367aa8_5446_423c_992a_2054e486fb65.slice/crio-eaaad47341d682dc504c98dda90bc8a4a4573e87bab40f8e3344ade52cb5c40b WatchSource:0}: Error finding container eaaad47341d682dc504c98dda90bc8a4a4573e87bab40f8e3344ade52cb5c40b: Status 404 returned error can't find the container with id eaaad47341d682dc504c98dda90bc8a4a4573e87bab40f8e3344ade52cb5c40b Oct 29 22:37:25.185322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.185300 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x"] Oct 29 22:37:25.187885 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.187850 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:25.187994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.187932 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:25.187994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.187978 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:25.319837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.319796 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" event={"ID":"b6367aa8-5446-423c-992a-2054e486fb65","Type":"ContainerStarted","Data":"eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995"} Oct 29 22:37:25.320047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.319854 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" event={"ID":"b6367aa8-5446-423c-992a-2054e486fb65","Type":"ContainerStarted","Data":"eaaad47341d682dc504c98dda90bc8a4a4573e87bab40f8e3344ade52cb5c40b"} Oct 29 22:37:25.371991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.371921 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podStartSLOduration=1.371901916 podStartE2EDuration="1.371901916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:25.364926577 +0000 UTC m=+1360.680320658" watchObservedRunningTime="2026-10-29 22:37:25.371901916 +0000 UTC m=+1360.687295998" Oct 29 22:37:25.827854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:25.827824 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:26.852608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:26.852577 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:27.337481 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:27.337452 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:27.338557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:27.338532 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:37:33.077615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:33.077579 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms"] Oct 29 22:37:33.099040 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:33.099011 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ce7be-qmwms"] Oct 29 22:37:34.839456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:34.839425 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e9f767-ead9-4d0c-a17e-29462ca243c6" path="/var/lib/kubelet/pods/a4e9f767-ead9-4d0c-a17e-29462ca243c6/volumes" Oct 29 22:37:39.255684 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:37:39.255639 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:37:47.987750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:47.987706 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb"] Oct 29 22:37:47.994338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:47.994316 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.015400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.015351 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb"] Oct 29 22:37:48.021413 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.021384 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-84304f-serving-cert\"" Oct 29 22:37:48.043686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.043652 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.044118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.044094 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.044348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.044331 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.044505 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.044491 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.145722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.145689 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.145918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.145787 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.145918 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:37:48.145848 5073 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-serving-cert: secret "maas-api-e2e-shared-b-w2-84304f-serving-cert" not found Oct 29 22:37:48.145918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.145874 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.146057 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:37:48.145941 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls podName:f1402654-ca43-41d9-afd6-94f7966c9783 nodeName:}" failed. No retries permitted until 2026-10-29 22:37:48.645920158 +0000 UTC m=+1383.961314220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls") pod "maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" (UID: "f1402654-ca43-41d9-afd6-94f7966c9783") : secret "maas-api-e2e-shared-b-w2-84304f-serving-cert" not found Oct 29 22:37:48.146121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.146103 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.146642 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.146622 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.148767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.148746 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.168681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.168650 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.654332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.654294 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.657457 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.657432 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.825274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.825221 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn"] Oct 29 22:37:48.831291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.831268 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.854178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.851227 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn"] Oct 29 22:37:48.856692 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.856673 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-6bb06a-openshift-default-dockercfg-vhrww\"" Oct 29 22:37:48.917480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.917403 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.962130 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.962466 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.962583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.963161 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.963242 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.963314 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.963366 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.963417 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:48.964177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:48.963464 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnpc\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065187 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065299 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065492 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065560 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065601 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065651 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065685 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065730 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnpc\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.066355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.065835 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.067287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.067142 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.069404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.067744 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.069404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.068297 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.069404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.068727 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.069404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.068936 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.070482 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.070268 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.075331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.074281 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.098189 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.098136 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.114206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.113894 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnpc\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.166258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.165954 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:49.384860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.383385 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb"] Oct 29 22:37:49.585605 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.585480 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" event={"ID":"f1402654-ca43-41d9-afd6-94f7966c9783","Type":"ContainerStarted","Data":"3f646c80e25e0aa7e664578c81e50e27f3162863e0369beab06d59b526cccc85"} Oct 29 22:37:49.800486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.800304 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn"] Oct 29 22:37:49.819966 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:37:49.819939 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd886ec9_e3ab_4b33_bf0b_3a574bcdf227.slice/crio-da2d0b73b1f6617f2aa4dce2cbaa1cbaca41869443a0fc6ae691d88236ee75d6 WatchSource:0}: Error finding container da2d0b73b1f6617f2aa4dce2cbaa1cbaca41869443a0fc6ae691d88236ee75d6: Status 404 returned error can't find the container with id da2d0b73b1f6617f2aa4dce2cbaa1cbaca41869443a0fc6ae691d88236ee75d6 Oct 29 22:37:49.824924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.824886 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:49.825039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.824958 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:49.825039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:49.824986 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:50.595960 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:50.595923 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" event={"ID":"f1402654-ca43-41d9-afd6-94f7966c9783","Type":"ContainerStarted","Data":"827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07"} Oct 29 22:37:50.596520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:50.596185 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:50.597991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:50.597966 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" event={"ID":"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227","Type":"ContainerStarted","Data":"3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2"} Oct 29 22:37:50.598097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:50.598010 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" event={"ID":"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227","Type":"ContainerStarted","Data":"da2d0b73b1f6617f2aa4dce2cbaa1cbaca41869443a0fc6ae691d88236ee75d6"} Oct 29 22:37:50.653177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:50.651528 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" podStartSLOduration=3.316778442 podStartE2EDuration="3.651507869s" totalImagesPullingTime="334.729427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:50.644366644 +0000 UTC m=+1385.959760748" watchObservedRunningTime="2026-10-29 22:37:50.651507869 +0000 UTC m=+1385.966901950" Oct 29 22:37:51.125920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.125863 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podStartSLOduration=3.125839489 podStartE2EDuration="3.125839489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:50.699446726 +0000 UTC m=+1386.014840831" watchObservedRunningTime="2026-10-29 22:37:51.125839489 +0000 UTC m=+1386.441233569" Oct 29 22:37:51.127251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.127231 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph"] Oct 29 22:37:51.131640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.131619 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.151807 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.149666 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f76fb5-openshift-default-dockercfg-lxhl2\"" Oct 29 22:37:51.168795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.168379 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:51.175306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.175282 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:51.181391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.181079 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph"] Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254626 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254676 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7sg\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254738 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254764 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254825 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254944 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.254979 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.255137 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.258170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.255196 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.369180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.369127 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.370026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.369988 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.374732 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.374713 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.375247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.375229 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.375828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.375809 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.385114 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.385060 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7sg\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.385736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.385715 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.385983 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.386082 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.375167 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386347 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.374535 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386347 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.369900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386560 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.380882 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.386818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.375600 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.389446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.389425 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.409712 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.408796 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.420595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.420531 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.455177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.438851 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7sg\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.471809 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.468426 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:51.621867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.619490 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:51.624687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.624658 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:37:51.901276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.901248 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph"] Oct 29 22:37:51.901748 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:37:51.901711 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda73d7b2_4e10_40aa_b1f6_a0837175b1a5.slice/crio-0ea1aebc67a980dc9ba3c6e619678c646bb18fa959a52ca7ad32d19f7a2f6478 WatchSource:0}: Error finding container 0ea1aebc67a980dc9ba3c6e619678c646bb18fa959a52ca7ad32d19f7a2f6478: Status 404 returned error can't find the container with id 0ea1aebc67a980dc9ba3c6e619678c646bb18fa959a52ca7ad32d19f7a2f6478 Oct 29 22:37:51.904571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.904538 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:51.904639 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.904601 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:51.904639 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:51.904629 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:52.627647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:52.627610 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" event={"ID":"da73d7b2-4e10-40aa-b1f6-a0837175b1a5","Type":"ContainerStarted","Data":"e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8"} Oct 29 22:37:52.628103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:52.627677 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" event={"ID":"da73d7b2-4e10-40aa-b1f6-a0837175b1a5","Type":"ContainerStarted","Data":"0ea1aebc67a980dc9ba3c6e619678c646bb18fa959a52ca7ad32d19f7a2f6478"} Oct 29 22:37:52.691068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:52.691007 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podStartSLOduration=1.6909880149999998 podStartE2EDuration="1.690988015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:52.68738622 +0000 UTC m=+1388.002780311" watchObservedRunningTime="2026-10-29 22:37:52.690988015 +0000 UTC m=+1388.006382093" Oct 29 22:37:53.082267 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:53.082234 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp"] Oct 29 22:37:53.102524 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:53.102493 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b24aa-z29zp"] Oct 29 22:37:53.506468 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:53.493180 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:53.512593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:53.512564 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:53.636333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:53.636295 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:53.637120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:53.637097 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:37:54.845497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:54.845288 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6a3fb1-f635-4eb3-ae36-29389a227fe8" path="/var/lib/kubelet/pods/fe6a3fb1-f635-4eb3-ae36-29389a227fe8/volumes" Oct 29 22:37:56.183907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.183873 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh"] Oct 29 22:37:56.190116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.190097 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.205657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.205633 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh"] Oct 29 22:37:56.218898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.218872 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-c7b96a-serving-cert\"" Oct 29 22:37:56.305070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.305034 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.305324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.305292 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xn89\" (UniqueName: \"kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.305407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.305324 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.305407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.305356 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.406863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.406826 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.407056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.406975 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xn89\" (UniqueName: \"kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.407056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.406995 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.407056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.407020 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.407880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.407855 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.411942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.410136 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.411942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.410217 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.429260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.429221 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xn89\" (UniqueName: \"kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89\") pod \"maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.525785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.525751 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:56.640110 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.640082 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 22:37:56.857024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:56.855287 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh"] Oct 29 22:37:56.862414 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:37:56.862368 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477ffde3_c6d7_4a62_9f90_6b6529e80df1.slice/crio-1ca668fb6029d012fa4beb1fbb793c79fcdb267cb51a1ddb59ed34767a198b72 WatchSource:0}: Error finding container 1ca668fb6029d012fa4beb1fbb793c79fcdb267cb51a1ddb59ed34767a198b72: Status 404 returned error can't find the container with id 1ca668fb6029d012fa4beb1fbb793c79fcdb267cb51a1ddb59ed34767a198b72 Oct 29 22:37:57.680440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:57.680344 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" event={"ID":"477ffde3-c6d7-4a62-9f90-6b6529e80df1","Type":"ContainerStarted","Data":"6fa68a2a0f2a12d9cc939db39cbbca33d8b1b2f059e60000f47352efd58c05d1"} Oct 29 22:37:57.680440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:57.680406 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" event={"ID":"477ffde3-c6d7-4a62-9f90-6b6529e80df1","Type":"ContainerStarted","Data":"1ca668fb6029d012fa4beb1fbb793c79fcdb267cb51a1ddb59ed34767a198b72"} Oct 29 22:37:57.680911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:57.680727 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:37:57.725843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:37:57.725789 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" podStartSLOduration=1.316614971 podStartE2EDuration="1.725776984s" totalImagesPullingTime="409.162013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:57.722100096 +0000 UTC m=+1393.037494176" watchObservedRunningTime="2026-10-29 22:37:57.725776984 +0000 UTC m=+1393.041171097" Oct 29 22:38:03.695410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:03.695373 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:38:09.257557 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:38:09.257515 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:38:16.023890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:16.023854 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:38:22.036052 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:22.036013 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px"] Oct 29 22:38:22.042377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:22.042353 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79736b-jx4px"] Oct 29 22:38:22.844021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:22.843988 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e3b60a-f86b-4e90-9fca-986cbf6371ab" path="/var/lib/kubelet/pods/b2e3b60a-f86b-4e90-9fca-986cbf6371ab/volumes" Oct 29 22:38:23.295136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.295099 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5"] Oct 29 22:38:23.301083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.301059 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.324818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.324788 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5"] Oct 29 22:38:23.382568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.382530 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0ab33e-serving-cert\"" Oct 29 22:38:23.475745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.475718 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.475878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.475757 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.475878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.475828 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.475878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.475875 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxvcq\" (UniqueName: \"kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.577514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.577427 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.577514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.577475 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.577722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.577564 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.577722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.577624 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxvcq\" (UniqueName: \"kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.578328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.578300 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.580481 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.580456 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.580650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.580539 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.600024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.599987 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxvcq\" (UniqueName: \"kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq\") pod \"maas-api-e2e-api-a-0ab33e-cb7577668-r76x5\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.627551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.627517 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:23.972507 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:38:23.972467 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b9921c_5661_4d4a_a0e9_acb326a5d976.slice/crio-c31a50b8bc5b97b8aa41657646c95f9a9932b774a0231b6570fc4eb3cebf2423 WatchSource:0}: Error finding container c31a50b8bc5b97b8aa41657646c95f9a9932b774a0231b6570fc4eb3cebf2423: Status 404 returned error can't find the container with id c31a50b8bc5b97b8aa41657646c95f9a9932b774a0231b6570fc4eb3cebf2423 Oct 29 22:38:23.973865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:23.973829 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5"] Oct 29 22:38:24.932553 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:24.931273 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" event={"ID":"29b9921c-5661-4d4a-a0e9-acb326a5d976","Type":"ContainerStarted","Data":"00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3"} Oct 29 22:38:24.932553 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:24.931341 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" event={"ID":"29b9921c-5661-4d4a-a0e9-acb326a5d976","Type":"ContainerStarted","Data":"c31a50b8bc5b97b8aa41657646c95f9a9932b774a0231b6570fc4eb3cebf2423"} Oct 29 22:38:24.932553 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:24.932491 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:24.982441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:24.982372 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" podStartSLOduration=1.576058486 podStartE2EDuration="1.98235163s" totalImagesPullingTime="406.293144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:24.98042308 +0000 UTC m=+1420.295817162" watchObservedRunningTime="2026-10-29 22:38:24.98235163 +0000 UTC m=+1420.297745710" Oct 29 22:38:25.778344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.778300 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh"] Oct 29 22:38:25.783666 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.783641 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.801531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.800610 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh"] Oct 29 22:38:25.814083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.813510 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-6bb06a-serving-cert\"" Oct 29 22:38:25.851026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.850984 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.851200 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.851059 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.851279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.851198 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lxb\" (UniqueName: \"kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.851423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.851373 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.953089 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.953056 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.953672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.953247 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.953672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.953300 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.953672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.953417 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lxb\" (UniqueName: \"kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.954142 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.954117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.956646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.956608 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.956646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.956634 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:25.974707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:25.974683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lxb\" (UniqueName: \"kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb\") pod \"maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:26.107763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:26.107671 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:26.463402 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:38:26.460345 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ed4fe7_ea1e_4f13_9bf1_f494f67459b3.slice/crio-32a6f49bae1d54d993a92ebd9bad9ceb8e07968e47dfa1f309268e5e15034ef6 WatchSource:0}: Error finding container 32a6f49bae1d54d993a92ebd9bad9ceb8e07968e47dfa1f309268e5e15034ef6: Status 404 returned error can't find the container with id 32a6f49bae1d54d993a92ebd9bad9ceb8e07968e47dfa1f309268e5e15034ef6 Oct 29 22:38:26.472676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:26.472608 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh"] Oct 29 22:38:26.951964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:26.951923 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" event={"ID":"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3","Type":"ContainerStarted","Data":"1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae"} Oct 29 22:38:26.952215 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:26.951995 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" event={"ID":"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3","Type":"ContainerStarted","Data":"32a6f49bae1d54d993a92ebd9bad9ceb8e07968e47dfa1f309268e5e15034ef6"} Oct 29 22:38:26.952215 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:26.952191 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:27.000611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.000553 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" podStartSLOduration=1.6634480200000001 podStartE2EDuration="2.000535485s" totalImagesPullingTime="337.087465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:26.996010128 +0000 UTC m=+1422.311404211" watchObservedRunningTime="2026-10-29 22:38:27.000535485 +0000 UTC m=+1422.315929566" Oct 29 22:38:27.918487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.916178 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": read tcp 10.132.0.2:46996->10.132.0.102:15021: read: connection reset by peer" start-of-body= Oct 29 22:38:27.918487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.916233 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": read tcp 10.132.0.2:46996->10.132.0.102:15021: read: connection reset by peer" Oct 29 22:38:27.965535 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.965498 5073 generic.go:356] "Generic (PLEG): container finished" podID="b6367aa8-5446-423c-992a-2054e486fb65" containerID="eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995" exitCode=0 Oct 29 22:38:27.965747 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.965573 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" event={"ID":"b6367aa8-5446-423c-992a-2054e486fb65","Type":"ContainerDied","Data":"eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995"} Oct 29 22:38:27.966434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.966418 5073 scope.go:122] "RemoveContainer" containerID="eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995" Oct 29 22:38:27.968466 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.968412 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:38:27.968587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.968541 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:38:27.968663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:27.968599 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:38:28.975857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:28.975824 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" event={"ID":"b6367aa8-5446-423c-992a-2054e486fb65","Type":"ContainerStarted","Data":"78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322"} Oct 29 22:38:29.837176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:29.836444 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:38:29.837176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:29.836824 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:38:29.837176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:29.836877 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:38:30.034122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.034087 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l"] Oct 29 22:38:30.039109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.039086 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.047918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.047527 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l"] Oct 29 22:38:30.056208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.056172 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f76fb5-serving-cert\"" Oct 29 22:38:30.061428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.061400 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.061552 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.061518 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.061616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.061565 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw62\" (UniqueName: \"kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.061670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.061610 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.162772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.162685 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.162772 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.162745 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw62\" (UniqueName: \"kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.163004 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.162787 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.163004 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:38:30.162848 5073 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-serving-cert: secret "maas-api-e2e-api-a-f76fb5-serving-cert" not found Oct 29 22:38:30.163004 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:38:30.162925 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls podName:b2f99906-78a7-4201-ac56-8c5d756301c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:38:30.662903293 +0000 UTC m=+1425.978297354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls") pod "maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" (UID: "b2f99906-78a7-4201-ac56-8c5d756301c0") : secret "maas-api-e2e-api-a-f76fb5-serving-cert" not found Oct 29 22:38:30.163205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.163114 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.163685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.163654 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.165917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.165893 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.189426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.189401 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw62\" (UniqueName: \"kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.675654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.675619 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.678504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.678477 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\") pod \"maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:30.833177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.831480 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:38:30.833177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.831534 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:38:30.958720 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:30.958695 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:31.306431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:31.306402 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l"] Oct 29 22:38:31.829340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:31.828693 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:38:31.829340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:31.828755 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:38:31.964330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:31.964303 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:38:32.012957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.008471 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" event={"ID":"b2f99906-78a7-4201-ac56-8c5d756301c0","Type":"ContainerStarted","Data":"ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91"} Oct 29 22:38:32.012957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.008538 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" event={"ID":"b2f99906-78a7-4201-ac56-8c5d756301c0","Type":"ContainerStarted","Data":"ea94e6c8f43405bb8d1034f5ac7eeaf22e4b05adae050ea50875bf876a23f17e"} Oct 29 22:38:32.026258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.019390 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:32.077226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.077168 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" podStartSLOduration=2.595779076 podStartE2EDuration="3.077131052s" totalImagesPullingTime="481.351976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:32.071494248 +0000 UTC m=+1427.386888328" watchObservedRunningTime="2026-10-29 22:38:32.077131052 +0000 UTC m=+1427.392525132" Oct 29 22:38:32.827809 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.827776 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:38:32.828404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.828373 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:38:32.973826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:32.973803 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 22:38:33.832176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:33.829563 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:38:33.832176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:33.829634 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:38:34.685890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:34.685844 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9"] Oct 29 22:38:34.686169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:34.686107 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" containerID="cri-o://f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab" gracePeriod=30 Oct 29 22:38:35.022502 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:35.022473 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:38:35.022965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:35.022806 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:38:35.023721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:35.023699 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:38:35.032407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:35.032385 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc"] Oct 29 22:38:35.036311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:35.036288 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7375-2z8bc"] Oct 29 22:38:36.850833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.850745 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06266413-4748-40d7-b64f-68b301ddbec4" path="/var/lib/kubelet/pods/06266413-4748-40d7-b64f-68b301ddbec4/volumes" Oct 29 22:38:36.854633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.851317 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z"] Oct 29 22:38:36.864509 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.864479 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:36.877018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.876967 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z"] Oct 29 22:38:36.882669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.882642 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d478b36e\"/\"kube-root-ca.crt\"" Oct 29 22:38:36.896550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.896428 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d478b36e\"/\"default-dockercfg-vhkl4\"" Oct 29 22:38:36.896550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.896459 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d478b36e\"/\"openshift-service-ca.crt\"" Oct 29 22:38:36.896550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.896511 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d478b36e\"/\"e2e-lifecycle-model-d478b36e-kserve-self-signed-certs\"" Oct 29 22:38:36.984852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.984817 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2q2b\" (UniqueName: \"kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:36.985042 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.984868 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:36.985355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.985323 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:36.985512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.985372 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:36.985512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.985427 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:36.985653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:36.985598 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.087261 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.087314 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.087381 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.087453 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.087549 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2q2b\" (UniqueName: \"kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.087585 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.088655 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.088896 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.093183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.090498 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.163248 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.151879 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.163248 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.157524 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.181203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.168817 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2q2b\" (UniqueName: \"kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b\") pod \"e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.199760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.199513 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:37.609316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:37.609278 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z"] Oct 29 22:38:37.609561 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:38:37.609531 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd246285c_98f5_4d1b_a35f_e279c5958921.slice/crio-f1280b2102094f409f0878a2d53350ffcfd05bd28137136fe8a958416e54babb WatchSource:0}: Error finding container f1280b2102094f409f0878a2d53350ffcfd05bd28137136fe8a958416e54babb: Status 404 returned error can't find the container with id f1280b2102094f409f0878a2d53350ffcfd05bd28137136fe8a958416e54babb Oct 29 22:38:38.170306 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:38.170268 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" event={"ID":"d246285c-98f5-4d1b-a35f-e279c5958921","Type":"ContainerStarted","Data":"fd6628d0b5b428199e1f0ac2a2775e4f08f9c59e7bf82d05db5af6857d7e9ad5"} Oct 29 22:38:38.170746 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:38.170318 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" event={"ID":"d246285c-98f5-4d1b-a35f-e279c5958921","Type":"ContainerStarted","Data":"f1280b2102094f409f0878a2d53350ffcfd05bd28137136fe8a958416e54babb"} Oct 29 22:38:38.200322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:38.200300 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:38.213269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:38.213224 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" podStartSLOduration=2.213211855 podStartE2EDuration="2.213211855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:38.211045723 +0000 UTC m=+1433.526439804" watchObservedRunningTime="2026-10-29 22:38:38.213211855 +0000 UTC m=+1433.528605967" Oct 29 22:38:39.075342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:39.075317 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:38:39.256817 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:38:39.256784 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:38:39.987354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:39.987325 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:38:40.147205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.141412 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf"] Oct 29 22:38:40.147205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.142299 5073 cpu_manager.go:414] "containerMap: removing container" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" Oct 29 22:38:40.147205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.142319 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" Oct 29 22:38:40.147205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.142478 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerName="istio-proxy" Oct 29 22:38:40.149210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149135 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149262 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149337 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149368 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149466 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149497 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149617 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149617 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149609 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.149720 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.149643 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token\") pod \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\" (UID: \"111ead0c-60ad-4fc1-8ac4-2ef6e56060db\") " Oct 29 22:38:40.151324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.151270 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.151611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.151590 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:40.152435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.152371 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.152435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.152408 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.152435 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.152420 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.152667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.152515 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.159295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.159263 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:40.159933 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.159903 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.169764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.168922 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.171001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.170977 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e526ec-openshift-default-dockercfg-rlvqd\"" Oct 29 22:38:40.186777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.186738 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l" pod "111ead0c-60ad-4fc1-8ac4-2ef6e56060db" (UID: "111ead0c-60ad-4fc1-8ac4-2ef6e56060db"). InnerVolumeSpecName "kube-api-access-nqs9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.217005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.216396 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf"] Oct 29 22:38:40.222020 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.221989 5073 generic.go:356] "Generic (PLEG): container finished" podID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" containerID="f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab" exitCode=0 Oct 29 22:38:40.222139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.222094 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" Oct 29 22:38:40.222139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.222121 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" event={"ID":"111ead0c-60ad-4fc1-8ac4-2ef6e56060db","Type":"ContainerDied","Data":"f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab"} Oct 29 22:38:40.222308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.222190 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9" event={"ID":"111ead0c-60ad-4fc1-8ac4-2ef6e56060db","Type":"ContainerDied","Data":"8e4bb23bc1a26ff8350eca88f866f9fb96a8ec35fdba46dcd0f48f29b518cc44"} Oct 29 22:38:40.222308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.222219 5073 scope.go:122] "RemoveContainer" containerID="f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab" Oct 29 22:38:40.237040 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.237021 5073 scope.go:122] "RemoveContainer" containerID="f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab" Oct 29 22:38:40.238236 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:38:40.237386 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab\": container with ID starting with f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab not found: ID does not exist" containerID="f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab" Oct 29 22:38:40.238236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.237419 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab"} err="failed to get container status \"f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab\": rpc error: code = NotFound desc = could not find container \"f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab\": container with ID starting with f7720b19556de4233b93c9d46c8d4f6be894f22602b802487db561f06e50e5ab not found: ID does not exist" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.252251 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.252326 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.252397 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5hv\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.252700 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.253234 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.253740 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.253848 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.253900 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254007 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254120 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254137 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254172 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254188 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254201 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254216 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254229 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqs9l\" (UniqueName: \"kubernetes.io/projected/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-kube-api-access-nqs9l\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254243 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.266177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.254255 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111ead0c-60ad-4fc1-8ac4-2ef6e56060db-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.297395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.297349 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9"] Oct 29 22:38:40.305834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.305802 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nfmw9"] Oct 29 22:38:40.355836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.355788 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356030 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356014 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356100 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356085 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356180 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5hv\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356237 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356321 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356369 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356399 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.356537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.356453 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.362559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.360126 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.362559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.360829 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.362559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.361712 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.363892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.363869 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.364327 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.364306 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.409170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.409005 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.411084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.411032 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.424133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.418849 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.435804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.433680 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5hv\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv\") pod \"e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.523277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.519197 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:40.858478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.858389 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111ead0c-60ad-4fc1-8ac4-2ef6e56060db" path="/var/lib/kubelet/pods/111ead0c-60ad-4fc1-8ac4-2ef6e56060db/volumes" Oct 29 22:38:40.974825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.966218 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:38:40.974825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.966312 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:38:40.974825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.966357 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:38:40.977770 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:40.977740 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf"] Oct 29 22:38:41.234802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:41.234766 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" event={"ID":"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda","Type":"ContainerStarted","Data":"cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb"} Oct 29 22:38:41.234967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:41.234825 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" event={"ID":"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda","Type":"ContainerStarted","Data":"ae3eb3d76b9d01d18ab16fd7f4b0bc49a187a048617fd0fced95cf7a87750ccc"} Oct 29 22:38:41.294841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:41.294775 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podStartSLOduration=1.294755576 podStartE2EDuration="1.294755576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:41.287753906 +0000 UTC m=+1436.603147988" watchObservedRunningTime="2026-10-29 22:38:41.294755576 +0000 UTC m=+1436.610149658" Oct 29 22:38:41.520786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:41.520696 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:42.526776 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:42.526748 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:43.255212 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:43.255185 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:43.256221 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:43.256199 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:38:45.424496 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:45.424467 5073 scope.go:122] "RemoveContainer" containerID="50d708a5e30e37dcaef9f55912e168167d1ff20828a96e24a64427cedaf64297" Oct 29 22:38:47.200134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:47.200099 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:47.214423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:47.214399 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:47.357064 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:47.357032 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:38:48.036272 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:48.036224 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s"] Oct 29 22:38:48.048838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:48.048814 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f47dde-tpz5s"] Oct 29 22:38:48.841066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:38:48.841025 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76efb132-ed2a-40d2-aab2-cf807aba073d" path="/var/lib/kubelet/pods/76efb132-ed2a-40d2-aab2-cf807aba073d/volumes" Oct 29 22:39:06.033101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:06.033067 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl"] Oct 29 22:39:06.049991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:06.049795 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365e54a5-99qfl"] Oct 29 22:39:06.843968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:06.843934 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6843d41f-752c-40c6-95e5-047992996920" path="/var/lib/kubelet/pods/6843d41f-752c-40c6-95e5-047992996920/volumes" Oct 29 22:39:07.665511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:07.665477 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:39:07.665948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:07.665545 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:08.637789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.637690 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:08.637789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.637764 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:08.656760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.656728 5073 generic.go:356] "Generic (PLEG): container finished" podID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerID="3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2" exitCode=0 Oct 29 22:39:08.656950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.656814 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" event={"ID":"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227","Type":"ContainerDied","Data":"3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2"} Oct 29 22:39:08.657542 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.657521 5073 scope.go:122] "RemoveContainer" containerID="3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2" Oct 29 22:39:08.670515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.670484 5073 generic.go:356] "Generic (PLEG): container finished" podID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerID="e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8" exitCode=0 Oct 29 22:39:08.670935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.670656 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" event={"ID":"da73d7b2-4e10-40aa-b1f6-a0837175b1a5","Type":"ContainerDied","Data":"e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8"} Oct 29 22:39:08.671284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.671250 5073 scope.go:122] "RemoveContainer" containerID="e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8" Oct 29 22:39:08.671861 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.671224 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:08.671988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.671961 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:08.672143 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.672091 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:08.674249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.674221 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:08.674373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.674352 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:08.674477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:08.674461 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:09.268204 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:39:09.266105 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:39:09.697062 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:09.685779 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" event={"ID":"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227","Type":"ContainerStarted","Data":"1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b"} Oct 29 22:39:09.708175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:09.707599 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" event={"ID":"da73d7b2-4e10-40aa-b1f6-a0837175b1a5","Type":"ContainerStarted","Data":"4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37"} Oct 29 22:39:10.167468 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:10.167430 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:39:10.167859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:10.167835 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:39:10.167929 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:10.167909 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:39:10.470392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:10.470357 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:10.470780 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:10.470748 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:10.471194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:10.470669 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:39:11.166913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:11.166869 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:39:11.167371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:11.166943 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:39:11.470452 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:11.470418 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:11.470653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:11.470482 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:12.167103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:12.167065 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.167705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:12.167118 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:39:12.470604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:12.470567 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.470886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:12.470858 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:13.172233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:13.169188 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.172233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:13.169250 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:39:13.481347 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:13.481315 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.481605 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:13.481578 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:14.168093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:14.168059 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.168395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:14.168366 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:39:14.472612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:14.472140 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.472612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:14.472217 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:15.188348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:15.177574 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.188348 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:15.177646 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:39:15.474529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:15.474494 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.479322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:15.479280 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:39:16.174755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:16.174720 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:39:16.176870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:16.176846 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:39:16.178411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:16.178378 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 22:39:16.474316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:16.474279 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:39:16.474653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:16.474628 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:39:16.475360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:16.475342 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:39:32.592079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:32.591975 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:39:39.256527 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:39:39.256490 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:39:45.047081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.047056 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:39:45.048818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.048793 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:39:45.052845 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.052821 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:39:45.055353 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.055334 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:39:45.462218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.462184 5073 scope.go:122] "RemoveContainer" containerID="787ac5de1d1e2bce3a2884a601b00f46ae5f7e373e1ef7ea9d8860a4b612d677" Oct 29 22:39:45.480419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.480390 5073 scope.go:122] "RemoveContainer" containerID="7f1a458e38ea0b83c649d7e3ca285022513668492b8624167d3d0239767abe86" Oct 29 22:39:45.494092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:45.494069 5073 scope.go:122] "RemoveContainer" containerID="3f55039a7fc6b7b3efc11880e3571d8e1f0e2563e7cdd58e22a8b73db9ce9d1e" Oct 29 22:39:49.324175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.321968 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8"] Oct 29 22:39:49.333215 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.333136 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.365126 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.365096 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:39:49.365358 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.365194 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:39:49.374362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.374140 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8"] Oct 29 22:39:49.422655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.422609 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.422865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.422723 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.423027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.423009 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxw9\" (UniqueName: \"kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.524548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.524517 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.525014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.524987 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxw9\" (UniqueName: \"kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.525191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.525133 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.525283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.525143 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.527644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.527621 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.546058 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.546036 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxw9\" (UniqueName: \"kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9\") pod \"maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.670458 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.670376 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:49.982256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.982225 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n"] Oct 29 22:39:49.982476 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:49.982452 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" containerID="cri-o://5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e" gracePeriod=30 Oct 29 22:39:50.018228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:50.018199 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8"] Oct 29 22:39:50.018595 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:39:50.018570 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6625267_1547_4165_b0ba_2d13c2e5ca27.slice/crio-a767854feb242134d3a3cf07021762098fa983eb0fa3c1e16d7fd462dd68c6cf WatchSource:0}: Error finding container a767854feb242134d3a3cf07021762098fa983eb0fa3c1e16d7fd462dd68c6cf: Status 404 returned error can't find the container with id a767854feb242134d3a3cf07021762098fa983eb0fa3c1e16d7fd462dd68c6cf Oct 29 22:39:50.552803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:50.552764 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" event={"ID":"f6625267-1547-4165-b0ba-2d13c2e5ca27","Type":"ContainerStarted","Data":"fdd76d093e58e6344c3b752624bcebba9cc23042582b163dad406e855942dfd6"} Oct 29 22:39:50.553215 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:50.552829 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" event={"ID":"f6625267-1547-4165-b0ba-2d13c2e5ca27","Type":"ContainerStarted","Data":"a767854feb242134d3a3cf07021762098fa983eb0fa3c1e16d7fd462dd68c6cf"} Oct 29 22:39:50.596235 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:50.596118 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" podStartSLOduration=1.5961037359999999 podStartE2EDuration="1.596103736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:50.59391837 +0000 UTC m=+1505.909312451" watchObservedRunningTime="2026-10-29 22:39:50.596103736 +0000 UTC m=+1505.911497873" Oct 29 22:39:51.565564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:51.565525 5073 generic.go:356] "Generic (PLEG): container finished" podID="f6625267-1547-4165-b0ba-2d13c2e5ca27" containerID="fdd76d093e58e6344c3b752624bcebba9cc23042582b163dad406e855942dfd6" exitCode=0 Oct 29 22:39:51.566038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:51.565572 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" event={"ID":"f6625267-1547-4165-b0ba-2d13c2e5ca27","Type":"ContainerDied","Data":"fdd76d093e58e6344c3b752624bcebba9cc23042582b163dad406e855942dfd6"} Oct 29 22:39:52.793000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.792976 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:52.880573 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.880498 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca\") pod \"f6625267-1547-4165-b0ba-2d13c2e5ca27\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " Oct 29 22:39:52.880709 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.880659 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9\" (UniqueName: \"kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9\") pod \"f6625267-1547-4165-b0ba-2d13c2e5ca27\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " Oct 29 22:39:52.880954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.880934 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca\") pod \"f6625267-1547-4165-b0ba-2d13c2e5ca27\" (UID: \"f6625267-1547-4165-b0ba-2d13c2e5ca27\") " Oct 29 22:39:52.881034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.880943 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca" pod "f6625267-1547-4165-b0ba-2d13c2e5ca27" (UID: "f6625267-1547-4165-b0ba-2d13c2e5ca27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:52.881888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.881838 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:52.883083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.883062 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9" pod "f6625267-1547-4165-b0ba-2d13c2e5ca27" (UID: "f6625267-1547-4165-b0ba-2d13c2e5ca27"). InnerVolumeSpecName "kube-api-access-xdxw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:52.883261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.883213 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca" pod "f6625267-1547-4165-b0ba-2d13c2e5ca27" (UID: "f6625267-1547-4165-b0ba-2d13c2e5ca27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:52.983232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.983210 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6625267-1547-4165-b0ba-2d13c2e5ca27-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:52.983345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:52.983234 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxw9\" (UniqueName: \"kubernetes.io/projected/f6625267-1547-4165-b0ba-2d13c2e5ca27-kube-api-access-xdxw9\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:53.581873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:53.581836 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" Oct 29 22:39:53.582071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:53.581824 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8" event={"ID":"f6625267-1547-4165-b0ba-2d13c2e5ca27","Type":"ContainerDied","Data":"a767854feb242134d3a3cf07021762098fa983eb0fa3c1e16d7fd462dd68c6cf"} Oct 29 22:39:53.582071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:53.581972 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a767854feb242134d3a3cf07021762098fa983eb0fa3c1e16d7fd462dd68c6cf" Oct 29 22:39:55.328311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.328279 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:39:55.409661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.409623 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g"] Oct 29 22:39:55.410531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.410510 5073 cpu_manager.go:414] "containerMap: removing container" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" Oct 29 22:39:55.410531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.410530 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" Oct 29 22:39:55.410685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.410545 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f6625267-1547-4165-b0ba-2d13c2e5ca27" containerName="revoke-keys" Oct 29 22:39:55.410685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.410552 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6625267-1547-4165-b0ba-2d13c2e5ca27" containerName="revoke-keys" Oct 29 22:39:55.410685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.410663 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c27486-7961-4706-afb7-2e0731289faf" containerName="istio-proxy" Oct 29 22:39:55.410685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.410671 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6625267-1547-4165-b0ba-2d13c2e5ca27" containerName="revoke-keys" Oct 29 22:39:55.414894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.414871 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.437018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.435850 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g"] Oct 29 22:39:55.519662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.519580 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.520537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520408 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.520665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520542 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.520665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520609 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.520665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520655 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.520878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520789 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.520931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520908 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.521041 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.520975 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.521041 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.521038 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert\") pod \"15c27486-7961-4706-afb7-2e0731289faf\" (UID: \"15c27486-7961-4706-afb7-2e0731289faf\") " Oct 29 22:39:55.521427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.521386 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.521718 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.521539 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.521718 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.521545 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527105 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527253 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527283 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527354 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527380 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.527800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527536 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq68n\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527602 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527673 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.527956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527869 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.527956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.527888 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.528329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.528301 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.528329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.528315 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.528485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.528375 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.538561 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.538523 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "kube-api-access-jpxbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.539079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.539040 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.539578 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.539552 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.539978 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.539943 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy" pod "15c27486-7961-4706-afb7-2e0731289faf" (UID: "15c27486-7961-4706-afb7-2e0731289faf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.602121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.602071 5073 generic.go:356] "Generic (PLEG): container finished" podID="15c27486-7961-4706-afb7-2e0731289faf" containerID="5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e" exitCode=0 Oct 29 22:39:55.602361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.602234 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" event={"ID":"15c27486-7961-4706-afb7-2e0731289faf","Type":"ContainerDied","Data":"5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e"} Oct 29 22:39:55.602361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.602292 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" event={"ID":"15c27486-7961-4706-afb7-2e0731289faf","Type":"ContainerDied","Data":"9ff38c890a3681b87f656ea0c23d5bcce79bd43ce45c78e907787b345d614b5e"} Oct 29 22:39:55.602361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.602324 5073 scope.go:122] "RemoveContainer" containerID="5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e" Oct 29 22:39:55.602543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.602496 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n" Oct 29 22:39:55.616615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.616594 5073 scope.go:122] "RemoveContainer" containerID="5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e" Oct 29 22:39:55.616929 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:39:55.616902 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e\": container with ID starting with 5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e not found: ID does not exist" containerID="5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e" Oct 29 22:39:55.617025 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.616941 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e"} err="failed to get container status \"5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e\": rpc error: code = NotFound desc = could not find container \"5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e\": container with ID starting with 5ea8a783bb5d9300b4ed6a96e961170325213b57390af31d08f371453e35908e not found: ID does not exist" Oct 29 22:39:55.629727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.629699 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.629936 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.629910 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630013 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630041 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630101 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630244 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq68n\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630307 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630341 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630500 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.630666 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630649 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c27486-7961-4706-afb7-2e0731289faf-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630675 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630690 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630705 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c27486-7961-4706-afb7-2e0731289faf-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630721 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c27486-7961-4706-afb7-2e0731289faf-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630736 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxbf\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-kube-api-access-jpxbf\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630750 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c27486-7961-4706-afb7-2e0731289faf-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.630955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.630852 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.631265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.631242 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.631528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.631497 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.631606 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.631544 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.631983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.631962 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.633705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.633669 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.634655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.634633 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.655963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.655933 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.656205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.656142 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq68n\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:55.662173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.662135 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n"] Oct 29 22:39:55.671888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.671865 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-lz69n"] Oct 29 22:39:55.744385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:55.744362 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:56.087109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.087080 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g"] Oct 29 22:39:56.092134 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:39:56.092094 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac646196_20b1_472d_832f_1f976df2bc78.slice/crio-62edea6502e21f5bf477894d3174f4e017ce211750c9aaf14f39a188f340bfa1 WatchSource:0}: Error finding container 62edea6502e21f5bf477894d3174f4e017ce211750c9aaf14f39a188f340bfa1: Status 404 returned error can't find the container with id 62edea6502e21f5bf477894d3174f4e017ce211750c9aaf14f39a188f340bfa1 Oct 29 22:39:56.095571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.095536 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:56.095652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.095606 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:56.095652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.095642 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:39:56.612793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.612754 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" event={"ID":"ac646196-20b1-472d-832f-1f976df2bc78","Type":"ContainerStarted","Data":"70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c"} Oct 29 22:39:56.613342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.612819 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" event={"ID":"ac646196-20b1-472d-832f-1f976df2bc78","Type":"ContainerStarted","Data":"62edea6502e21f5bf477894d3174f4e017ce211750c9aaf14f39a188f340bfa1"} Oct 29 22:39:56.657706 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.657660 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podStartSLOduration=5.657645459 podStartE2EDuration="5.657645459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:56.656489853 +0000 UTC m=+1511.971883933" watchObservedRunningTime="2026-10-29 22:39:56.657645459 +0000 UTC m=+1511.973039618" Oct 29 22:39:56.745165 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.745123 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:39:56.845956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:56.845912 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c27486-7961-4706-afb7-2e0731289faf" path="/var/lib/kubelet/pods/15c27486-7961-4706-afb7-2e0731289faf/volumes" Oct 29 22:39:57.746319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:57.746278 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:57.746845 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:57.746358 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:39:58.749470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:58.748615 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:58.749470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:58.748706 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:39:59.745232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:59.745202 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:59.745418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:39:59.745265 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:40:00.747599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:00.747565 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:40:00.748244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:00.748210 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:40:01.765180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:01.763662 5073 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.765180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:01.763733 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:40:02.771053 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:02.763232 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d"] Oct 29 22:40:02.787963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:02.782492 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.078234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.012117 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1ae775\"" Oct 29 22:40:03.078234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.039806 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.131187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.129686 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1ae775-dockercfg-nxnzl\"" Oct 29 22:40:03.161122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.161088 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhj4h\" (UniqueName: \"kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h\") pod \"payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.259567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.259532 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d"] Oct 29 22:40:03.259927 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.259905 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:40:03.261217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.261199 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:40:03.265311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.265097 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.265536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.265427 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:40:03.265889 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.265850 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhj4h\" (UniqueName: \"kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h\") pod \"payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.266440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.266411 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.287492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.287460 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhj4h\" (UniqueName: \"kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h\") pod \"payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.499924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.499892 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:03.839186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.839118 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d"] Oct 29 22:40:03.840475 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:40:03.840447 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3522e3_86c9_4dcb_877f_61e47fbcf8b5.slice/crio-a11c5a9657da92cf25ea1581a4f6696a45e167adb6255fd16ce125b42020db8e WatchSource:0}: Error finding container a11c5a9657da92cf25ea1581a4f6696a45e167adb6255fd16ce125b42020db8e: Status 404 returned error can't find the container with id a11c5a9657da92cf25ea1581a4f6696a45e167adb6255fd16ce125b42020db8e Oct 29 22:40:03.842628 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:03.842604 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:40:04.850282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:04.850245 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" event={"ID":"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5","Type":"ContainerStarted","Data":"a11c5a9657da92cf25ea1581a4f6696a45e167adb6255fd16ce125b42020db8e"} Oct 29 22:40:07.888188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:07.888131 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" event={"ID":"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5","Type":"ContainerStarted","Data":"edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462"} Oct 29 22:40:07.888589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:07.888228 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:07.931472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:07.931330 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" podStartSLOduration=2.537984379 podStartE2EDuration="5.931310642s" totalImagesPullingTime="3.393326263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:07.928774177 +0000 UTC m=+1523.244168258" watchObservedRunningTime="2026-10-29 22:40:07.931310642 +0000 UTC m=+1523.246704735" Oct 29 22:40:09.256963 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:40:09.256919 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:40:11.925543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:11.925509 5073 generic.go:356] "Generic (PLEG): container finished" podID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerID="cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb" exitCode=0 Oct 29 22:40:11.926120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:11.925578 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" event={"ID":"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda","Type":"ContainerDied","Data":"cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb"} Oct 29 22:40:11.926120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:11.926017 5073 scope.go:122] "RemoveContainer" containerID="cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb" Oct 29 22:40:11.927841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:11.927807 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:40:11.927952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:11.927877 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:40:11.927952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:11.927910 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:40:12.952211 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:12.952113 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" event={"ID":"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda","Type":"ContainerStarted","Data":"cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204"} Oct 29 22:40:13.256097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:13.256057 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:40:13.524572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:13.524484 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:40:13.534313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:13.534279 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:40:13.534568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:13.534541 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:40:14.535176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:14.534921 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:40:14.535176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:14.534985 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:40:15.522610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:15.522574 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:40:15.522897 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:15.522869 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:40:16.528593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:16.528559 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:40:16.533338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:16.533292 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:40:17.523179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:17.522479 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:40:17.523179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:17.522536 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:40:18.546331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:18.546293 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:40:18.548349 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:18.548319 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:40:18.900777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:18.900702 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:40:25.996614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:25.996580 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:40:39.257442 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:40:39.257395 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:40:45.534963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:45.534922 5073 scope.go:122] "RemoveContainer" containerID="1f22baa8109f5e62e2ef6d6a13733aa3e337078ee2bb944546107329a17af636" Oct 29 22:40:45.548517 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:40:45.548496 5073 scope.go:122] "RemoveContainer" containerID="d15408db911e31a6bb1c4180e2a0890e3d06c52b1d21330fd5170ae570e637ac" Oct 29 22:41:09.258400 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:41:09.258312 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:41:16.360341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:41:16.360306 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8"] Oct 29 22:41:16.373565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:41:16.373538 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d478b36e-w2gw8"] Oct 29 22:41:16.839921 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:41:16.839887 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6625267-1547-4165-b0ba-2d13c2e5ca27" path="/var/lib/kubelet/pods/f6625267-1547-4165-b0ba-2d13c2e5ca27/volumes" Oct 29 22:41:38.465297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:41:38.465260 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:41:39.264526 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:41:39.264483 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:42:07.684612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.684580 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt"] Oct 29 22:42:07.689272 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.689249 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:07.707340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.707312 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt"] Oct 29 22:42:07.715364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.715336 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:07.715540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.715384 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:42:07.892172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.889334 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:07.892172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.889497 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:07.892172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:07.889575 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvk7\" (UniqueName: \"kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.025177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.002660 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.025662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.025632 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.025910 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.025888 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvk7\" (UniqueName: \"kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.038735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.017527 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.038958 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.026840 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.080237 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.075443 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvk7\" (UniqueName: \"kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7\") pod \"maas-api-revoke-sub-73734f08-f29qt\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.304888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.304787 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:08.619924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:08.619889 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt"] Oct 29 22:42:08.621855 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:08.621831 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35659891_4589_473e_b226_ba3fe55c328f.slice/crio-2bb7519a1dcc5f7ddd98719cc9d603bdbf2ae3dd4d01f359eaabeb2ad2ec52d0 WatchSource:0}: Error finding container 2bb7519a1dcc5f7ddd98719cc9d603bdbf2ae3dd4d01f359eaabeb2ad2ec52d0: Status 404 returned error can't find the container with id 2bb7519a1dcc5f7ddd98719cc9d603bdbf2ae3dd4d01f359eaabeb2ad2ec52d0 Oct 29 22:42:09.166097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:09.166061 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" event={"ID":"35659891-4589-473e-b226-ba3fe55c328f","Type":"ContainerStarted","Data":"06fca2069b3e14c6a7eae225070cb75a4be9fe47ff7635a397972f5e2683c215"} Oct 29 22:42:09.166609 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:09.166118 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" event={"ID":"35659891-4589-473e-b226-ba3fe55c328f","Type":"ContainerStarted","Data":"2bb7519a1dcc5f7ddd98719cc9d603bdbf2ae3dd4d01f359eaabeb2ad2ec52d0"} Oct 29 22:42:09.213392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:09.213333 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" podStartSLOduration=2.213314682 podStartE2EDuration="2.213314682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:09.207702802 +0000 UTC m=+1644.523096882" watchObservedRunningTime="2026-10-29 22:42:09.213314682 +0000 UTC m=+1644.528708765" Oct 29 22:42:09.257565 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:09.257534 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:42:10.176847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:10.176811 5073 generic.go:356] "Generic (PLEG): container finished" podID="35659891-4589-473e-b226-ba3fe55c328f" containerID="06fca2069b3e14c6a7eae225070cb75a4be9fe47ff7635a397972f5e2683c215" exitCode=0 Oct 29 22:42:10.177356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:10.177327 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" event={"ID":"35659891-4589-473e-b226-ba3fe55c328f","Type":"ContainerDied","Data":"06fca2069b3e14c6a7eae225070cb75a4be9fe47ff7635a397972f5e2683c215"} Oct 29 22:42:11.371692 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.371649 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:11.519601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.519573 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca\") pod \"35659891-4589-473e-b226-ba3fe55c328f\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " Oct 29 22:42:11.520207 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.520184 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7\" (UniqueName: \"kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7\") pod \"35659891-4589-473e-b226-ba3fe55c328f\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " Oct 29 22:42:11.520416 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.520395 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca\") pod \"35659891-4589-473e-b226-ba3fe55c328f\" (UID: \"35659891-4589-473e-b226-ba3fe55c328f\") " Oct 29 22:42:11.520818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.520075 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca" pod "35659891-4589-473e-b226-ba3fe55c328f" (UID: "35659891-4589-473e-b226-ba3fe55c328f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:11.521821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.521797 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35659891-4589-473e-b226-ba3fe55c328f-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:11.522900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.522875 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7" pod "35659891-4589-473e-b226-ba3fe55c328f" (UID: "35659891-4589-473e-b226-ba3fe55c328f"). InnerVolumeSpecName "kube-api-access-hpvk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:11.523343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.523321 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca" pod "35659891-4589-473e-b226-ba3fe55c328f" (UID: "35659891-4589-473e-b226-ba3fe55c328f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:11.623865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.623836 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvk7\" (UniqueName: \"kubernetes.io/projected/35659891-4589-473e-b226-ba3fe55c328f-kube-api-access-hpvk7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:11.623865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:11.623861 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35659891-4589-473e-b226-ba3fe55c328f-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.196071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:12.196039 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" Oct 29 22:42:12.196326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:12.196075 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt" event={"ID":"35659891-4589-473e-b226-ba3fe55c328f","Type":"ContainerDied","Data":"2bb7519a1dcc5f7ddd98719cc9d603bdbf2ae3dd4d01f359eaabeb2ad2ec52d0"} Oct 29 22:42:12.196326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:12.196132 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bb7519a1dcc5f7ddd98719cc9d603bdbf2ae3dd4d01f359eaabeb2ad2ec52d0" Oct 29 22:42:32.345031 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:32.339997 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:42:38.685296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:38.685262 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5"] Oct 29 22:42:38.685954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:38.685514 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" podUID="29b9921c-5661-4d4a-a0e9-acb326a5d976" containerName="maas-api" containerID="cri-o://00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3" gracePeriod=30 Oct 29 22:42:39.393120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.378368 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:42:39.423175 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:39.415866 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:42:39.474612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.474057 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47"] Oct 29 22:42:39.475695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.475666 5073 cpu_manager.go:414] "containerMap: removing container" podUID="29b9921c-5661-4d4a-a0e9-acb326a5d976" containerName="maas-api" Oct 29 22:42:39.475695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.475695 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b9921c-5661-4d4a-a0e9-acb326a5d976" containerName="maas-api" Oct 29 22:42:39.476749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.475720 5073 cpu_manager.go:414] "containerMap: removing container" podUID="35659891-4589-473e-b226-ba3fe55c328f" containerName="revoke-keys" Oct 29 22:42:39.476749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.475728 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35659891-4589-473e-b226-ba3fe55c328f" containerName="revoke-keys" Oct 29 22:42:39.476749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.476144 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b9921c-5661-4d4a-a0e9-acb326a5d976" containerName="maas-api" Oct 29 22:42:39.476749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.476187 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="35659891-4589-473e-b226-ba3fe55c328f" containerName="revoke-keys" Oct 29 22:42:39.483491 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.483458 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls\") pod \"29b9921c-5661-4d4a-a0e9-acb326a5d976\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " Oct 29 22:42:39.483491 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.483487 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.483670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.483616 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq\" (UniqueName: \"kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq\") pod \"29b9921c-5661-4d4a-a0e9-acb326a5d976\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " Oct 29 22:42:39.483738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.483715 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca\") pod \"29b9921c-5661-4d4a-a0e9-acb326a5d976\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " Oct 29 22:42:39.483804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.483783 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs\") pod \"29b9921c-5661-4d4a-a0e9-acb326a5d976\" (UID: \"29b9921c-5661-4d4a-a0e9-acb326a5d976\") " Oct 29 22:42:39.485833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.485799 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca" pod "29b9921c-5661-4d4a-a0e9-acb326a5d976" (UID: "29b9921c-5661-4d4a-a0e9-acb326a5d976"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:39.489479 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.489455 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs" pod "29b9921c-5661-4d4a-a0e9-acb326a5d976" (UID: "29b9921c-5661-4d4a-a0e9-acb326a5d976"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:39.505169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.503829 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47"] Oct 29 22:42:39.505169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.504496 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls" pod "29b9921c-5661-4d4a-a0e9-acb326a5d976" (UID: "29b9921c-5661-4d4a-a0e9-acb326a5d976"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:39.509709 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.508580 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-433712-openshift-default-dockercfg-lnlp8\"" Oct 29 22:42:39.520224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.520195 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq" pod "29b9921c-5661-4d4a-a0e9-acb326a5d976" (UID: "29b9921c-5661-4d4a-a0e9-acb326a5d976"). InnerVolumeSpecName "kube-api-access-kxvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:39.523734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.523703 5073 generic.go:356] "Generic (PLEG): container finished" podID="29b9921c-5661-4d4a-a0e9-acb326a5d976" containerID="00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3" exitCode=0 Oct 29 22:42:39.524083 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.524059 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" Oct 29 22:42:39.524607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.524572 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" event={"ID":"29b9921c-5661-4d4a-a0e9-acb326a5d976","Type":"ContainerDied","Data":"00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3"} Oct 29 22:42:39.524837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.524817 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5" event={"ID":"29b9921c-5661-4d4a-a0e9-acb326a5d976","Type":"ContainerDied","Data":"c31a50b8bc5b97b8aa41657646c95f9a9932b774a0231b6570fc4eb3cebf2423"} Oct 29 22:42:39.524967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.524859 5073 scope.go:122] "RemoveContainer" containerID="00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3" Oct 29 22:42:39.565790 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.565588 5073 scope.go:122] "RemoveContainer" containerID="00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3" Oct 29 22:42:39.566208 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:39.566140 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3\": container with ID starting with 00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3 not found: ID does not exist" containerID="00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3" Oct 29 22:42:39.566309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.566220 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3"} err="failed to get container status \"00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3\": rpc error: code = NotFound desc = could not find container \"00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3\": container with ID starting with 00fcf23619a77d96d5cda3f6a06597b22e950a73e74094cc94c98be7d3326ef3 not found: ID does not exist" Oct 29 22:42:39.586052 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586019 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnfh\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586066 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586225 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586457 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586431 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586551 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586628 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586613 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586642 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586725 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586773 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.586939 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586926 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.586987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586945 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxvcq\" (UniqueName: \"kubernetes.io/projected/29b9921c-5661-4d4a-a0e9-acb326a5d976-kube-api-access-kxvcq\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.586987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586960 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b9921c-5661-4d4a-a0e9-acb326a5d976-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.586987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.586973 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b9921c-5661-4d4a-a0e9-acb326a5d976-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.609233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.609207 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5"] Oct 29 22:42:39.613156 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.613126 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0ab33e-cb7577668-r76x5"] Oct 29 22:42:39.688027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.687996 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnfh\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688042 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688129 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688299 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688396 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688436 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688454 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688501 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688533 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688689 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.688858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688793 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.689011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688877 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.689011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.688951 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.689516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.689491 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.690775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.690755 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.690898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.690880 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.709163 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.709128 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.709442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.709425 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnfh\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-dsl47\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:39.860652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:39.859136 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:40.210577 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.210547 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47"] Oct 29 22:42:40.211007 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:40.210968 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a5e002_8dc8_47c4_a9fb_6c54fa5a8d8f.slice/crio-78b7d210eb1be036e858cb5dad79e85254c6e868832ba02fc8cf400ffe65a378 WatchSource:0}: Error finding container 78b7d210eb1be036e858cb5dad79e85254c6e868832ba02fc8cf400ffe65a378: Status 404 returned error can't find the container with id 78b7d210eb1be036e858cb5dad79e85254c6e868832ba02fc8cf400ffe65a378 Oct 29 22:42:40.213728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.213696 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:40.213819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.213759 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:40.213819 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.213792 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:40.534292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.534192 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" event={"ID":"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f","Type":"ContainerStarted","Data":"9ebda31ffd63d8bfe696d163459acee1b872ab449d528f929e60a117bd7ce1a0"} Oct 29 22:42:40.534292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.534259 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" event={"ID":"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f","Type":"ContainerStarted","Data":"78b7d210eb1be036e858cb5dad79e85254c6e868832ba02fc8cf400ffe65a378"} Oct 29 22:42:40.583382 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.583337 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podStartSLOduration=94.583324394 podStartE2EDuration="1m34.583324394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:40.581303054 +0000 UTC m=+1675.896697136" watchObservedRunningTime="2026-10-29 22:42:40.583324394 +0000 UTC m=+1675.898718473" Oct 29 22:42:40.844896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.844799 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b9921c-5661-4d4a-a0e9-acb326a5d976" path="/var/lib/kubelet/pods/29b9921c-5661-4d4a-a0e9-acb326a5d976/volumes" Oct 29 22:42:40.859718 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:40.859694 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:41.861053 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:41.861017 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-dsl47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:42:41.861527 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:41.861070 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:42:42.860241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:42.860210 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-dsl47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:42:42.860420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:42.860269 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:42:43.860388 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:43.860352 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-dsl47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:42:43.860787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:43.860408 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:42:44.476353 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:44.476324 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l"] Oct 29 22:42:44.476567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:44.476526 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" podUID="b2f99906-78a7-4201-ac56-8c5d756301c0" containerName="maas-api" containerID="cri-o://ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91" gracePeriod=30 Oct 29 22:42:45.051644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.051611 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-dsl47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:42:45.052230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.052196 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:42:45.474752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.474355 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:42:45.542220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.540532 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\") pod \"b2f99906-78a7-4201-ac56-8c5d756301c0\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " Oct 29 22:42:45.542220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.540606 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62\" (UniqueName: \"kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62\") pod \"b2f99906-78a7-4201-ac56-8c5d756301c0\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " Oct 29 22:42:45.542220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.540729 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca\") pod \"b2f99906-78a7-4201-ac56-8c5d756301c0\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " Oct 29 22:42:45.542220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.540809 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs\") pod \"b2f99906-78a7-4201-ac56-8c5d756301c0\" (UID: \"b2f99906-78a7-4201-ac56-8c5d756301c0\") " Oct 29 22:42:45.547345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.547298 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca" pod "b2f99906-78a7-4201-ac56-8c5d756301c0" (UID: "b2f99906-78a7-4201-ac56-8c5d756301c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:45.557299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.557268 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs" pod "b2f99906-78a7-4201-ac56-8c5d756301c0" (UID: "b2f99906-78a7-4201-ac56-8c5d756301c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:45.593538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.593491 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls" pod "b2f99906-78a7-4201-ac56-8c5d756301c0" (UID: "b2f99906-78a7-4201-ac56-8c5d756301c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:45.594992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.594951 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62" pod "b2f99906-78a7-4201-ac56-8c5d756301c0" (UID: "b2f99906-78a7-4201-ac56-8c5d756301c0"). InnerVolumeSpecName "kube-api-access-cqw62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:45.607501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.607475 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb"] Oct 29 22:42:45.645117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.643115 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:45.645117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.643172 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqw62\" (UniqueName: \"kubernetes.io/projected/b2f99906-78a7-4201-ac56-8c5d756301c0-kube-api-access-cqw62\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:45.645117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.643188 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f99906-78a7-4201-ac56-8c5d756301c0-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:45.645117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.643202 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f99906-78a7-4201-ac56-8c5d756301c0-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:45.655872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.650430 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2f99906-78a7-4201-ac56-8c5d756301c0" containerName="maas-api" Oct 29 22:42:45.655872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.650453 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f99906-78a7-4201-ac56-8c5d756301c0" containerName="maas-api" Oct 29 22:42:45.655872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.650714 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f99906-78a7-4201-ac56-8c5d756301c0" containerName="maas-api" Oct 29 22:42:45.678473 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.676412 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2"] Oct 29 22:42:45.681512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.679255 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:45.704397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.704359 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2"] Oct 29 22:42:45.704797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.704778 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb"] Oct 29 22:42:45.705095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.705082 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:45.772127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.771547 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-63db1a-dockercfg-84stk\"" Oct 29 22:42:45.772127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.771704 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-63db1a\"" Oct 29 22:42:45.785302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.781837 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2f99906-78a7-4201-ac56-8c5d756301c0" containerID="ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91" exitCode=0 Oct 29 22:42:45.785302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.782045 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" event={"ID":"b2f99906-78a7-4201-ac56-8c5d756301c0","Type":"ContainerDied","Data":"ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91"} Oct 29 22:42:45.785302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.782102 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" event={"ID":"b2f99906-78a7-4201-ac56-8c5d756301c0","Type":"ContainerDied","Data":"ea94e6c8f43405bb8d1034f5ac7eeaf22e4b05adae050ea50875bf876a23f17e"} Oct 29 22:42:45.785302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.782135 5073 scope.go:122] "RemoveContainer" containerID="ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91" Oct 29 22:42:45.785302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.783380 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l" Oct 29 22:42:45.825248 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.824237 5073 scope.go:122] "RemoveContainer" containerID="ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91" Oct 29 22:42:45.827351 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:45.827325 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91\": container with ID starting with ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91 not found: ID does not exist" containerID="ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91" Oct 29 22:42:45.827497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.827471 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91"} err="failed to get container status \"ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91\": rpc error: code = NotFound desc = could not find container \"ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91\": container with ID starting with ca93dde7e9c5baad2fa237301addd29c4c2970a436dcf14b9148734169dd8e91 not found: ID does not exist" Oct 29 22:42:45.860679 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.860651 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-dsl47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:42:45.860777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.860694 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:42:45.868699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.868673 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l"] Oct 29 22:42:45.878606 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.878574 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76fb5-84bf4b45f8-mh98l"] Oct 29 22:42:45.880625 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.880596 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbcb\" (UniqueName: \"kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb\") pod \"payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:45.880734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.880675 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95w7t\" (UniqueName: \"kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t\") pod \"payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:45.880734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.880721 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:45.882629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.882600 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume\") pod \"payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:45.984235 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.984207 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbcb\" (UniqueName: \"kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb\") pod \"payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:45.984379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.984260 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95w7t\" (UniqueName: \"kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t\") pod \"payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:45.984379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.984291 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:45.984454 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.984436 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume\") pod \"payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:45.985056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.985040 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume\") pod \"payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:45.985093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:45.985061 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:46.005377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.005352 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95w7t\" (UniqueName: \"kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t\") pod \"payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:46.005494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.005432 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbcb\" (UniqueName: \"kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb\") pod \"payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:46.103386 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.103312 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:46.132174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.132122 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:46.441166 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:46.440700 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cbb1be_56f4_4b03_84ba_c05d7d909594.slice/crio-100ae690c692b12fa96d2407d9a3d8b9258ff02db040cd003ddb78f6c98a03b4 WatchSource:0}: Error finding container 100ae690c692b12fa96d2407d9a3d8b9258ff02db040cd003ddb78f6c98a03b4: Status 404 returned error can't find the container with id 100ae690c692b12fa96d2407d9a3d8b9258ff02db040cd003ddb78f6c98a03b4 Oct 29 22:42:46.442055 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.441929 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb"] Oct 29 22:42:46.449420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.449396 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2"] Oct 29 22:42:46.450616 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:46.450590 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5576976_85ba_42c5_80f2_f2801ab2d9f9.slice/crio-e2ee90665a2fb770fa14ab06f96eff4f0fd2cfd29afb01aef59bc528c4f85129 WatchSource:0}: Error finding container e2ee90665a2fb770fa14ab06f96eff4f0fd2cfd29afb01aef59bc528c4f85129: Status 404 returned error can't find the container with id e2ee90665a2fb770fa14ab06f96eff4f0fd2cfd29afb01aef59bc528c4f85129 Oct 29 22:42:46.794214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.794172 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" event={"ID":"a5576976-85ba-42c5-80f2-f2801ab2d9f9","Type":"ContainerStarted","Data":"8e21f927469684a036a8f306837aaf7ff4058dbd27fd508b1c9992a78276ed4a"} Oct 29 22:42:46.794420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.794239 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" event={"ID":"a5576976-85ba-42c5-80f2-f2801ab2d9f9","Type":"ContainerStarted","Data":"e2ee90665a2fb770fa14ab06f96eff4f0fd2cfd29afb01aef59bc528c4f85129"} Oct 29 22:42:46.794420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.794324 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:46.796202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.796134 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" event={"ID":"55cbb1be-56f4-4b03-84ba-c05d7d909594","Type":"ContainerStarted","Data":"52fab272e7ae74cd742151b463f25cb36ccc48e61a2531d7478c262208ae7886"} Oct 29 22:42:46.796305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.796217 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" event={"ID":"55cbb1be-56f4-4b03-84ba-c05d7d909594","Type":"ContainerStarted","Data":"100ae690c692b12fa96d2407d9a3d8b9258ff02db040cd003ddb78f6c98a03b4"} Oct 29 22:42:46.796305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.796242 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:46.836697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.836652 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" podStartSLOduration=65.836636872 podStartE2EDuration="1m5.836636872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:46.834752866 +0000 UTC m=+1682.150146960" watchObservedRunningTime="2026-10-29 22:42:46.836636872 +0000 UTC m=+1682.152030952" Oct 29 22:42:46.841982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.841957 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f99906-78a7-4201-ac56-8c5d756301c0" path="/var/lib/kubelet/pods/b2f99906-78a7-4201-ac56-8c5d756301c0/volumes" Oct 29 22:42:46.865158 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.865127 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:46.865411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.865389 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:46.866113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.866094 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:42:46.876021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:46.875984 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" podStartSLOduration=65.875972423 podStartE2EDuration="1m5.875972423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:46.875263184 +0000 UTC m=+1682.190657264" watchObservedRunningTime="2026-10-29 22:42:46.875972423 +0000 UTC m=+1682.191366502" Oct 29 22:42:47.960671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:47.960640 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt"] Oct 29 22:42:47.966491 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:47.966469 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:47.983174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:47.983135 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt"] Oct 29 22:42:47.992343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:47.992313 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:47.992482 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:47.992380 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:42:48.036515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.036477 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.036727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.036706 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjchg\" (UniqueName: \"kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.036799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.036782 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.138298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.138267 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjchg\" (UniqueName: \"kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.139032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.138380 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.139032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.138522 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.139273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.139235 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.141625 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.141605 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.158562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.158538 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjchg\" (UniqueName: \"kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg\") pod \"maas-api-revoke-keys-e2e-api-b-433712-x8hgt\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.282829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.282778 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:48.585885 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:48.585850 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6c99d4_38b9_474a_a852_f14057e908eb.slice/crio-b3fa02d06a564a8aea3fe33acdc8467baeda22cdd48d85c9a9bca514452c5e97 WatchSource:0}: Error finding container b3fa02d06a564a8aea3fe33acdc8467baeda22cdd48d85c9a9bca514452c5e97: Status 404 returned error can't find the container with id b3fa02d06a564a8aea3fe33acdc8467baeda22cdd48d85c9a9bca514452c5e97 Oct 29 22:42:48.586125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.586103 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt"] Oct 29 22:42:48.820597 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:48.820566 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" event={"ID":"fa6c99d4-38b9-474a-a852-f14057e908eb","Type":"ContainerStarted","Data":"b3fa02d06a564a8aea3fe33acdc8467baeda22cdd48d85c9a9bca514452c5e97"} Oct 29 22:42:49.829823 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:49.829748 5073 generic.go:356] "Generic (PLEG): container finished" podID="fa6c99d4-38b9-474a-a852-f14057e908eb" containerID="0056318871de6a11c15f8c96ff3376a612cb84ab735e4b4c2ad8441126c427bc" exitCode=0 Oct 29 22:42:49.830237 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:49.829809 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" event={"ID":"fa6c99d4-38b9-474a-a852-f14057e908eb","Type":"ContainerDied","Data":"0056318871de6a11c15f8c96ff3376a612cb84ab735e4b4c2ad8441126c427bc"} Oct 29 22:42:51.031963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.031936 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:51.091397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.091333 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg\" (UniqueName: \"kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg\") pod \"fa6c99d4-38b9-474a-a852-f14057e908eb\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " Oct 29 22:42:51.091558 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.091543 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca\") pod \"fa6c99d4-38b9-474a-a852-f14057e908eb\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " Oct 29 22:42:51.091693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.091679 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca\") pod \"fa6c99d4-38b9-474a-a852-f14057e908eb\" (UID: \"fa6c99d4-38b9-474a-a852-f14057e908eb\") " Oct 29 22:42:51.092099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.092068 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca" pod "fa6c99d4-38b9-474a-a852-f14057e908eb" (UID: "fa6c99d4-38b9-474a-a852-f14057e908eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:51.092466 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.092442 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:51.094077 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.094055 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca" pod "fa6c99d4-38b9-474a-a852-f14057e908eb" (UID: "fa6c99d4-38b9-474a-a852-f14057e908eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:51.094391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.094346 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg" pod "fa6c99d4-38b9-474a-a852-f14057e908eb" (UID: "fa6c99d4-38b9-474a-a852-f14057e908eb"). InnerVolumeSpecName "kube-api-access-tjchg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:51.194591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.194538 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6c99d4-38b9-474a-a852-f14057e908eb-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:51.194591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.194576 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjchg\" (UniqueName: \"kubernetes.io/projected/fa6c99d4-38b9-474a-a852-f14057e908eb-kube-api-access-tjchg\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:51.849513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.849485 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" Oct 29 22:42:51.849672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.849521 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt" event={"ID":"fa6c99d4-38b9-474a-a852-f14057e908eb","Type":"ContainerDied","Data":"b3fa02d06a564a8aea3fe33acdc8467baeda22cdd48d85c9a9bca514452c5e97"} Oct 29 22:42:51.849672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:51.849574 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3fa02d06a564a8aea3fe33acdc8467baeda22cdd48d85c9a9bca514452c5e97" Oct 29 22:42:53.170079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.170038 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g"] Oct 29 22:42:53.170622 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.170262 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" containerID="cri-o://70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c" gracePeriod=30 Oct 29 22:42:53.182918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.182890 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph"] Oct 29 22:42:53.183427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.183360 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" containerID="cri-o://4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37" gracePeriod=30 Oct 29 22:42:53.818010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.817978 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4"] Oct 29 22:42:53.818725 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.818708 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fa6c99d4-38b9-474a-a852-f14057e908eb" containerName="revoke-keys" Oct 29 22:42:53.818725 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.818725 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6c99d4-38b9-474a-a852-f14057e908eb" containerName="revoke-keys" Oct 29 22:42:53.818909 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.818879 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6c99d4-38b9-474a-a852-f14057e908eb" containerName="revoke-keys" Oct 29 22:42:53.822790 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.822771 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.840785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.840764 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4"] Oct 29 22:42:53.851665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.851638 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:53.851775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.851754 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:42:53.891717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.891658 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.891863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.891841 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.892006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.891987 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjp25\" (UniqueName: \"kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.993510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.993481 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.993624 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.993554 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.993680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.993633 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjp25\" (UniqueName: \"kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.994167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.994133 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:53.996026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:53.996008 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:54.014679 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:54.014654 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjp25\" (UniqueName: \"kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25\") pod \"maas-api-revoke-keys-e2e-api-b-c06e05-z76k4\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:54.136818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:54.136749 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:54.463420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:54.460406 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4"] Oct 29 22:42:54.464729 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:54.463495 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5998404b_22e0_4ea9_b8dc_fd940e9337fe.slice/crio-ca571033a4512ac3d8b3b315fd71aace25c5d33c8112c214fb321ce9593e3ae4 WatchSource:0}: Error finding container ca571033a4512ac3d8b3b315fd71aace25c5d33c8112c214fb321ce9593e3ae4: Status 404 returned error can't find the container with id ca571033a4512ac3d8b3b315fd71aace25c5d33c8112c214fb321ce9593e3ae4 Oct 29 22:42:54.877977 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:54.877924 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" event={"ID":"5998404b-22e0-4ea9-b8dc-fd940e9337fe","Type":"ContainerStarted","Data":"ca571033a4512ac3d8b3b315fd71aace25c5d33c8112c214fb321ce9593e3ae4"} Oct 29 22:42:55.888539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:55.888510 5073 generic.go:356] "Generic (PLEG): container finished" podID="5998404b-22e0-4ea9-b8dc-fd940e9337fe" containerID="2ab366051914f7a57accbb70acea4c8dd568f81c00601c30ac064df709e9e8e1" exitCode=0 Oct 29 22:42:55.888912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:55.888584 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" event={"ID":"5998404b-22e0-4ea9-b8dc-fd940e9337fe","Type":"ContainerDied","Data":"2ab366051914f7a57accbb70acea4c8dd568f81c00601c30ac064df709e9e8e1"} Oct 29 22:42:57.096098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.096071 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:57.240350 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.240322 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca\") pod \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " Oct 29 22:42:57.240528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.240452 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca\") pod \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " Oct 29 22:42:57.240528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.240483 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25\" (UniqueName: \"kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25\") pod \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\" (UID: \"5998404b-22e0-4ea9-b8dc-fd940e9337fe\") " Oct 29 22:42:57.240691 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.240668 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca" pod "5998404b-22e0-4ea9-b8dc-fd940e9337fe" (UID: "5998404b-22e0-4ea9-b8dc-fd940e9337fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:57.240976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.240955 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.242686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.242659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca" pod "5998404b-22e0-4ea9-b8dc-fd940e9337fe" (UID: "5998404b-22e0-4ea9-b8dc-fd940e9337fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:57.242846 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.242821 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25" pod "5998404b-22e0-4ea9-b8dc-fd940e9337fe" (UID: "5998404b-22e0-4ea9-b8dc-fd940e9337fe"). InnerVolumeSpecName "kube-api-access-kjp25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:57.342653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.342627 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5998404b-22e0-4ea9-b8dc-fd940e9337fe-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.342653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.342650 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjp25\" (UniqueName: \"kubernetes.io/projected/5998404b-22e0-4ea9-b8dc-fd940e9337fe-kube-api-access-kjp25\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.817993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.817966 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:42:57.818140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.818031 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:42:57.912664 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.912641 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" Oct 29 22:42:57.912838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.912763 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4" event={"ID":"5998404b-22e0-4ea9-b8dc-fd940e9337fe","Type":"ContainerDied","Data":"ca571033a4512ac3d8b3b315fd71aace25c5d33c8112c214fb321ce9593e3ae4"} Oct 29 22:42:57.912838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:57.912811 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca571033a4512ac3d8b3b315fd71aace25c5d33c8112c214fb321ce9593e3ae4" Oct 29 22:42:58.481593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.481565 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:42:58.512654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.512603 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.550300 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc"] Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.550978 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5998404b-22e0-4ea9-b8dc-fd940e9337fe" containerName="revoke-keys" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.550990 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5998404b-22e0-4ea9-b8dc-fd940e9337fe" containerName="revoke-keys" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551028 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551033 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551057 5073 cpu_manager.go:414] "containerMap: removing container" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551062 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551072 5073 cpu_manager.go:414] "containerMap: removing container" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551077 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551206 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5998404b-22e0-4ea9-b8dc-fd940e9337fe" containerName="revoke-keys" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551219 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac646196-20b1-472d-832f-1f976df2bc78" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551229 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" Oct 29 22:42:58.552256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.551235 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerName="istio-proxy" Oct 29 22:42:58.555136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.555112 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.563500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563473 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563584 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563640 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563679 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563729 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563763 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563795 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.563952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.563932 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.564132 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.564110 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token\") pod \"ac646196-20b1-472d-832f-1f976df2bc78\" (UID: \"ac646196-20b1-472d-832f-1f976df2bc78\") " Oct 29 22:42:58.565339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.564106 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.565339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.564421 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.565339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.564967 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.565339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.565278 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.565339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.565298 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.565339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.565312 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.565841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.565808 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:58.565969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.565880 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.573608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.573582 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.573683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.573659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:58.573717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.573677 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "kube-api-access-sq68n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:58.573749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.573713 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token" pod "ac646196-20b1-472d-832f-1f976df2bc78" (UID: "ac646196-20b1-472d-832f-1f976df2bc78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:58.576615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.576593 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc"] Oct 29 22:42:58.590633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.590609 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5"] Oct 29 22:42:58.595393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.595372 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.617194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.616467 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-139eb7-openshift-default-dockercfg-zq8wj\"" Oct 29 22:42:58.620835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.620813 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5"] Oct 29 22:42:58.667996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.667719 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.667996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.667833 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.667996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.667868 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.667996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.667916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.668379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.668002 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.668379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.668040 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.668379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.668130 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.668379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.668227 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.672240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.671227 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert\") pod \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\" (UID: \"da73d7b2-4e10-40aa-b1f6-a0837175b1a5\") " Oct 29 22:42:58.672240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.671670 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.672240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.671775 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.672240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.671808 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5x6\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.672240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672094 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.672614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672309 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.672614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672396 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.672614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672495 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.672614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672521 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.672614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672592 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672616 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672639 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672678 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672711 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672808 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672839 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.672866 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnd8c\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673008 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673077 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673216 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq68n\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-kube-api-access-sq68n\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673232 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac646196-20b1-472d-832f-1f976df2bc78-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673246 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac646196-20b1-472d-832f-1f976df2bc78-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673259 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673270 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac646196-20b1-472d-832f-1f976df2bc78-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.673282 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac646196-20b1-472d-832f-1f976df2bc78-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.675546 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.679900 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.680075 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.680950 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.683116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.681120 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.684460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.684328 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:58.689253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.688286 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:58.689253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.688403 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "kube-api-access-bn7sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:58.690103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.690079 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo" pod "da73d7b2-4e10-40aa-b1f6-a0837175b1a5" (UID: "da73d7b2-4e10-40aa-b1f6-a0837175b1a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:58.774529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774490 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774538 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774553 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774595 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774613 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774631 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.774695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774695 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774770 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774837 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774872 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnd8c\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.774984 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775016 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775052 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775078 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775121 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775140 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775363 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775377 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775421 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5x6\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775538 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775552 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.775683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775614 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.775683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.775647 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.776140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776078 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.776140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776104 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776136 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776397 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7sg\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-kube-api-access-bn7sg\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776416 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776431 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776462 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776476 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776490 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776505 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.776576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776530 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da73d7b2-4e10-40aa-b1f6-a0837175b1a5-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.777048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.776983 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.777322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.777298 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.778138 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.778117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.778292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.778133 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.778292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.778164 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.778593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.778575 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.796893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.796827 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.798192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.797305 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnd8c\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.798337 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.798261 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5x6\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-5b5qc\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.798543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.798521 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.876320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.876294 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:42:58.912821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.912797 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:42:58.923067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.923038 5073 generic.go:356] "Generic (PLEG): container finished" podID="ac646196-20b1-472d-832f-1f976df2bc78" containerID="70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c" exitCode=0 Oct 29 22:42:58.923278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.923080 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" Oct 29 22:42:58.923278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.923115 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" event={"ID":"ac646196-20b1-472d-832f-1f976df2bc78","Type":"ContainerDied","Data":"70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c"} Oct 29 22:42:58.923278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.923218 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g" event={"ID":"ac646196-20b1-472d-832f-1f976df2bc78","Type":"ContainerDied","Data":"62edea6502e21f5bf477894d3174f4e017ce211750c9aaf14f39a188f340bfa1"} Oct 29 22:42:58.923278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.923248 5073 scope.go:122] "RemoveContainer" containerID="70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c" Oct 29 22:42:58.925511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.925488 5073 generic.go:356] "Generic (PLEG): container finished" podID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" containerID="4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37" exitCode=0 Oct 29 22:42:58.925621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.925519 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" event={"ID":"da73d7b2-4e10-40aa-b1f6-a0837175b1a5","Type":"ContainerDied","Data":"4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37"} Oct 29 22:42:58.925621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.925580 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" event={"ID":"da73d7b2-4e10-40aa-b1f6-a0837175b1a5","Type":"ContainerDied","Data":"0ea1aebc67a980dc9ba3c6e619678c646bb18fa959a52ca7ad32d19f7a2f6478"} Oct 29 22:42:58.925951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.925667 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph" Oct 29 22:42:58.936967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.936925 5073 scope.go:122] "RemoveContainer" containerID="70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c" Oct 29 22:42:58.937286 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:58.937262 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c\": container with ID starting with 70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c not found: ID does not exist" containerID="70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c" Oct 29 22:42:58.937405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.937382 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c"} err="failed to get container status \"70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c\": rpc error: code = NotFound desc = could not find container \"70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c\": container with ID starting with 70f02ddd3e0d2529e95658205418bebd5538db864210520216e76606835df91c not found: ID does not exist" Oct 29 22:42:58.937405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.937408 5073 scope.go:122] "RemoveContainer" containerID="4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37" Oct 29 22:42:58.972206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.959379 5073 scope.go:122] "RemoveContainer" containerID="e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8" Oct 29 22:42:58.989470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.989368 5073 scope.go:122] "RemoveContainer" containerID="4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37" Oct 29 22:42:58.990226 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:58.989859 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37\": container with ID starting with 4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37 not found: ID does not exist" containerID="4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37" Oct 29 22:42:58.990226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.989923 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37"} err="failed to get container status \"4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37\": rpc error: code = NotFound desc = could not find container \"4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37\": container with ID starting with 4180da9dcc0c18594a862eab7be0f75a109df149b00a3018f53e1671292aab37 not found: ID does not exist" Oct 29 22:42:58.990226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.989948 5073 scope.go:122] "RemoveContainer" containerID="e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8" Oct 29 22:42:58.990746 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:42:58.990708 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8\": container with ID starting with e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8 not found: ID does not exist" containerID="e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8" Oct 29 22:42:58.990847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.990747 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8"} err="failed to get container status \"e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8\": rpc error: code = NotFound desc = could not find container \"e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8\": container with ID starting with e7368e60592e60249204f2cfaeccc8fb9addbe5cf5a3d0c33062cb9544fb28a8 not found: ID does not exist" Oct 29 22:42:58.993968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:58.993946 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph"] Oct 29 22:42:59.021857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.021831 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-wsqph"] Oct 29 22:42:59.043082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.043010 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g"] Oct 29 22:42:59.061988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.061935 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-b2m5g"] Oct 29 22:42:59.335745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.335654 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc"] Oct 29 22:42:59.370916 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:59.370863 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb856c23_72ff_4d6e_8e2b_6255e5c8b0e4.slice/crio-339b13b724b7beb08e47f819ebaf0bc5d356923f477abc560de8168879df9b18 WatchSource:0}: Error finding container 339b13b724b7beb08e47f819ebaf0bc5d356923f477abc560de8168879df9b18: Status 404 returned error can't find the container with id 339b13b724b7beb08e47f819ebaf0bc5d356923f477abc560de8168879df9b18 Oct 29 22:42:59.378003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.376740 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:59.378003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.376837 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:59.378003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.376878 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:59.488212 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.488180 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5"] Oct 29 22:42:59.497237 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:42:59.497208 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790bb1ce_002a_4745_b825_467d8872aaec.slice/crio-c3ed213a07ab21e512bb74c668d55f57d269d06975c9859fced364e86f6f8a0e WatchSource:0}: Error finding container c3ed213a07ab21e512bb74c668d55f57d269d06975c9859fced364e86f6f8a0e: Status 404 returned error can't find the container with id c3ed213a07ab21e512bb74c668d55f57d269d06975c9859fced364e86f6f8a0e Oct 29 22:42:59.500764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.500733 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:59.500827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.500799 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:59.500860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.500830 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:42:59.943977 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.943879 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" event={"ID":"790bb1ce-002a-4745-b825-467d8872aaec","Type":"ContainerStarted","Data":"a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f"} Oct 29 22:42:59.943977 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.943942 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" event={"ID":"790bb1ce-002a-4745-b825-467d8872aaec","Type":"ContainerStarted","Data":"c3ed213a07ab21e512bb74c668d55f57d269d06975c9859fced364e86f6f8a0e"} Oct 29 22:42:59.946397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.946299 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" event={"ID":"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4","Type":"ContainerStarted","Data":"6f6b5bd8ac0ab23bd5ff7f54f419fe5f2baefb61d0c43581718cd1d776125f19"} Oct 29 22:42:59.946397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:42:59.946358 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" event={"ID":"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4","Type":"ContainerStarted","Data":"339b13b724b7beb08e47f819ebaf0bc5d356923f477abc560de8168879df9b18"} Oct 29 22:43:00.005455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.005402 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podStartSLOduration=68.005388699 podStartE2EDuration="1m8.005388699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:00.002473606 +0000 UTC m=+1695.317867697" watchObservedRunningTime="2026-10-29 22:43:00.005388699 +0000 UTC m=+1695.320782778" Oct 29 22:43:00.052293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.052246 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podStartSLOduration=69.05223242 podStartE2EDuration="1m9.05223242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:00.047551869 +0000 UTC m=+1695.362945947" watchObservedRunningTime="2026-10-29 22:43:00.05223242 +0000 UTC m=+1695.367626499" Oct 29 22:43:00.853055 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.853027 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac646196-20b1-472d-832f-1f976df2bc78" path="/var/lib/kubelet/pods/ac646196-20b1-472d-832f-1f976df2bc78/volumes" Oct 29 22:43:00.854061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.854035 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da73d7b2-4e10-40aa-b1f6-a0837175b1a5" path="/var/lib/kubelet/pods/da73d7b2-4e10-40aa-b1f6-a0837175b1a5/volumes" Oct 29 22:43:00.877494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.877465 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:43:00.878908 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.878884 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:43:00.879051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.878931 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 29 22:43:00.924565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.924488 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:43:00.940582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.940554 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:43:00.957278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.957249 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:43:00.958784 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:00.958733 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:43:01.877336 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:01.877297 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:43:01.877739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:01.877347 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 29 22:43:02.879289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:02.879251 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:43:02.879726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:02.879309 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 29 22:43:03.877228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:03.877136 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:43:03.877432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:03.877274 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 29 22:43:04.880481 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:04.880444 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:43:04.880970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:04.880940 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 29 22:43:05.715358 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:05.715305 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2"] Oct 29 22:43:05.715890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:05.715842 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" containerID="cri-o://8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38" gracePeriod=30 Oct 29 22:43:05.883246 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:05.883211 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:43:05.883722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:05.883667 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:43:05.884501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:05.884485 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:43:07.996343 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:07.996303 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:43:07.996800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:07.996376 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:09.267456 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:43:09.266128 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:43:11.015644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.015619 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:43:11.095844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.095813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.095844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.095857 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.095917 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096079 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096113 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096165 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096208 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096330 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096459 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs\") pod \"093fcca4-bf99-43c2-83ed-a757047f52e0\" (UID: \"093fcca4-bf99-43c2-83ed-a757047f52e0\") " Oct 29 22:43:11.096836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.096803 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:11.102351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.097284 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093fcca4-bf99-43c2-83ed-a757047f52e0-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.102351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.097515 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.102351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.098342 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.102351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.098412 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.102351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.100718 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:11.102351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.101998 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.103745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.103719 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "kube-api-access-jgb5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:11.104903 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.104873 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:11.105784 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.105758 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy" pod "093fcca4-bf99-43c2-83ed-a757047f52e0" (UID: "093fcca4-bf99-43c2-83ed-a757047f52e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.107346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.107323 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm"] Oct 29 22:43:11.108053 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.108035 5073 cpu_manager.go:414] "containerMap: removing container" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" Oct 29 22:43:11.108139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.108056 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" Oct 29 22:43:11.108278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.108265 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerName="istio-proxy" Oct 29 22:43:11.112456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.112437 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.125965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.125707 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm"] Oct 29 22:43:11.129621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.129585 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d478b36e-openshift-default-dockercfg-6hzxg\"" Oct 29 22:43:11.130322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.130299 5073 generic.go:356] "Generic (PLEG): container finished" podID="093fcca4-bf99-43c2-83ed-a757047f52e0" containerID="8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38" exitCode=0 Oct 29 22:43:11.130426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.130341 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" Oct 29 22:43:11.130426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.130377 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" event={"ID":"093fcca4-bf99-43c2-83ed-a757047f52e0","Type":"ContainerDied","Data":"8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38"} Oct 29 22:43:11.130504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.130427 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2" event={"ID":"093fcca4-bf99-43c2-83ed-a757047f52e0","Type":"ContainerDied","Data":"abb7c468f52604ac331c5ee53befb79c2bc105f35fcdaa08b68ea822c82b58cb"} Oct 29 22:43:11.130504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.130456 5073 scope.go:122] "RemoveContainer" containerID="8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38" Oct 29 22:43:11.154007 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.153321 5073 scope.go:122] "RemoveContainer" containerID="8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38" Oct 29 22:43:11.154007 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:43:11.153641 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38\": container with ID starting with 8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38 not found: ID does not exist" containerID="8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38" Oct 29 22:43:11.154007 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.153675 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38"} err="failed to get container status \"8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38\": rpc error: code = NotFound desc = could not find container \"8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38\": container with ID starting with 8ad50383398d59ee4e48b55a9933cb9d52e51e1ccd4f1f8c16a958bcb2808b38 not found: ID does not exist" Oct 29 22:43:11.207346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.207310 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.207549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.207378 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.207549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.207459 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.207549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.207492 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfvv\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.208023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.207995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.208169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208096 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208267 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208357 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208521 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208668 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208686 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208699 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208712 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208725 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208737 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgb5f\" (UniqueName: \"kubernetes.io/projected/093fcca4-bf99-43c2-83ed-a757047f52e0-kube-api-access-jgb5f\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208749 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.208951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.208764 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093fcca4-bf99-43c2-83ed-a757047f52e0-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.215424 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.215395 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2"] Oct 29 22:43:11.219565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.219538 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-9rsx2"] Oct 29 22:43:11.310520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310446 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310519 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310698 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310638 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310702 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310801 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310852 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310896 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.310982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.310966 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.311043 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.311011 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfvv\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.311093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.311073 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.311201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.311135 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.311501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.311437 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.311501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.311445 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.311501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.311448 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.313078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.313058 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.313514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.313498 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.332130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.332110 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.332376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.332358 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfvv\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.462066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.462047 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:11.785061 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:43:11.785028 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711f9cff_d484_4ad7_a198_848987f51d17.slice/crio-4d47d55e4d75e873376b1c68e512e99970628e48f5c0185ce5558ab5c9b6fba5 WatchSource:0}: Error finding container 4d47d55e4d75e873376b1c68e512e99970628e48f5c0185ce5558ab5c9b6fba5: Status 404 returned error can't find the container with id 4d47d55e4d75e873376b1c68e512e99970628e48f5c0185ce5558ab5c9b6fba5 Oct 29 22:43:11.785442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.785404 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm"] Oct 29 22:43:11.788023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.787989 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:11.788103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.788059 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:11.788103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:11.788092 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:12.139839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.139752 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" event={"ID":"711f9cff-d484-4ad7-a198-848987f51d17","Type":"ContainerStarted","Data":"36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562"} Oct 29 22:43:12.139839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.139805 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" event={"ID":"711f9cff-d484-4ad7-a198-848987f51d17","Type":"ContainerStarted","Data":"4d47d55e4d75e873376b1c68e512e99970628e48f5c0185ce5558ab5c9b6fba5"} Oct 29 22:43:12.212818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.212765 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podStartSLOduration=3.212750698 podStartE2EDuration="3.212750698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:12.198523175 +0000 UTC m=+1707.513917265" watchObservedRunningTime="2026-10-29 22:43:12.212750698 +0000 UTC m=+1707.528144776" Oct 29 22:43:12.462826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.462801 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:12.464091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.464068 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:43:12.464206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.464113 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 29 22:43:12.843596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:12.843524 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093fcca4-bf99-43c2-83ed-a757047f52e0" path="/var/lib/kubelet/pods/093fcca4-bf99-43c2-83ed-a757047f52e0/volumes" Oct 29 22:43:13.469188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:13.469130 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:43:13.469782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:13.469751 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 29 22:43:14.462822 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:14.462784 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:43:14.463046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:14.462844 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 29 22:43:15.570177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:15.569277 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:43:15.570177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:15.569344 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 29 22:43:16.462414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:16.462377 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:43:16.462568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:16.462453 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 29 22:43:17.462783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:17.462745 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:43:17.463200 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:17.462811 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 29 22:43:18.615115 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:18.615050 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:18.615658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:18.615578 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:18.617366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:18.617339 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:43:29.289938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:29.289888 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d"] Oct 29 22:43:34.290944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.290905 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" podUID="ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" containerName="payload-pre-processing" containerID="cri-o://edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462" gracePeriod=25 Oct 29 22:43:34.577892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.577868 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:43:34.691049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.691021 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume\") pod \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " Oct 29 22:43:34.691224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.691208 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h\" (UniqueName: \"kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h\") pod \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\" (UID: \"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5\") " Oct 29 22:43:34.691388 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.691367 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume" pod "ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" (UID: "ca3522e3-86c9-4dcb-877f-61e47fbcf8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:34.691826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.691800 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:34.693515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.693490 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h" pod "ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" (UID: "ca3522e3-86c9-4dcb-877f-61e47fbcf8b5"). InnerVolumeSpecName "kube-api-access-fhj4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:34.793103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:34.793076 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhj4h\" (UniqueName: \"kubernetes.io/projected/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5-kube-api-access-fhj4h\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:35.331744 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.331710 5073 generic.go:356] "Generic (PLEG): container finished" podID="ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" containerID="edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462" exitCode=0 Oct 29 22:43:35.332111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.331759 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" Oct 29 22:43:35.332111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.331808 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" event={"ID":"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5","Type":"ContainerDied","Data":"edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462"} Oct 29 22:43:35.332111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.331865 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d" event={"ID":"ca3522e3-86c9-4dcb-877f-61e47fbcf8b5","Type":"ContainerDied","Data":"a11c5a9657da92cf25ea1581a4f6696a45e167adb6255fd16ce125b42020db8e"} Oct 29 22:43:35.332111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.331893 5073 scope.go:122] "RemoveContainer" containerID="edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462" Oct 29 22:43:35.342721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.342691 5073 scope.go:122] "RemoveContainer" containerID="edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462" Oct 29 22:43:35.342998 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:43:35.342979 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462\": container with ID starting with edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462 not found: ID does not exist" containerID="edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462" Oct 29 22:43:35.343070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.343007 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462"} err="failed to get container status \"edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462\": rpc error: code = NotFound desc = could not find container \"edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462\": container with ID starting with edfc2a8bb7a296ed0ac5599d110d058ceae6d330cb283a75377dfccaa034c462 not found: ID does not exist" Oct 29 22:43:35.388446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.388424 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d"] Oct 29 22:43:35.401783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:35.401759 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ae775-65879db4ff-k9q8d"] Oct 29 22:43:36.839831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:36.839800 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" path="/var/lib/kubelet/pods/ca3522e3-86c9-4dcb-877f-61e47fbcf8b5/volumes" Oct 29 22:43:39.256007 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:43:39.255970 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:43:50.306195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:50.306137 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw"] Oct 29 22:43:50.306782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:50.306410 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" containerID="cri-o://137ff07fcf9251bd0648c3f440399af83a57bceb9612cce43a498fe6209ebe45" gracePeriod=30 Oct 29 22:43:51.055693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:51.055653 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt"] Oct 29 22:43:51.055883 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:51.055864 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" containerID="cri-o://f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d" gracePeriod=30 Oct 29 22:43:55.209354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.209279 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:43:55.564810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.564779 5073 generic.go:356] "Generic (PLEG): container finished" podID="0b7fbd69-c165-41e7-a768-2443557073ff" containerID="137ff07fcf9251bd0648c3f440399af83a57bceb9612cce43a498fe6209ebe45" exitCode=0 Oct 29 22:43:55.564937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.564857 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" event={"ID":"0b7fbd69-c165-41e7-a768-2443557073ff","Type":"ContainerDied","Data":"137ff07fcf9251bd0648c3f440399af83a57bceb9612cce43a498fe6209ebe45"} Oct 29 22:43:55.564985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.564941 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" event={"ID":"0b7fbd69-c165-41e7-a768-2443557073ff","Type":"ContainerDied","Data":"012059c1b144d3393ca03d7d93441661b895b3db85d60d54e58c827599438b11"} Oct 29 22:43:55.564985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.564970 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012059c1b144d3393ca03d7d93441661b895b3db85d60d54e58c827599438b11" Oct 29 22:43:55.587688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.587669 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:43:55.670779 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.670749 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj"] Oct 29 22:43:55.672233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.672208 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" containerName="payload-pre-processing" Oct 29 22:43:55.672233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.672231 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" containerName="payload-pre-processing" Oct 29 22:43:55.672419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.672273 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" Oct 29 22:43:55.672419 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.672282 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" Oct 29 22:43:55.672521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.672467 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3522e3-86c9-4dcb-877f-61e47fbcf8b5" containerName="payload-pre-processing" Oct 29 22:43:55.672521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.672484 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" containerName="istio-proxy" Oct 29 22:43:55.676837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.676808 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.693953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.693924 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj"] Oct 29 22:43:55.773644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773583 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773625 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773655 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773703 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773796 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773771 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773869 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773944 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.773988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.773970 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.774137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774093 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.774137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774117 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:55.774302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774269 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.774367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774350 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.774474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774458 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp\") pod \"0b7fbd69-c165-41e7-a768-2443557073ff\" (UID: \"0b7fbd69-c165-41e7-a768-2443557073ff\") " Oct 29 22:43:55.774610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774585 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.774776 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774758 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.774896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774880 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775004 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.774984 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775044 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775089 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775135 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775120 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775437 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775414 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775606 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775576 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775701 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775625 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfnb\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.775783 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775773 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.775857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775791 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.775857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775804 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fbd69-c165-41e7-a768-2443557073ff-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.775857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775833 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.776000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.775859 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.778209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.778119 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:55.779233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.779196 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:55.779349 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.779290 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "kube-api-access-h24qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:55.779528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.779501 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token" pod "0b7fbd69-c165-41e7-a768-2443557073ff" (UID: "0b7fbd69-c165-41e7-a768-2443557073ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:55.878270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878240 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878300 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878328 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878355 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878373 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878495 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878624 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfnb\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878716 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878749 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878781 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878832 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.878922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878884 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fbd69-c165-41e7-a768-2443557073ff-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.878922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878904 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h24qp\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-kube-api-access-h24qp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.878922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878919 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fbd69-c165-41e7-a768-2443557073ff-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.879111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.878934 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fbd69-c165-41e7-a768-2443557073ff-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:55.879111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.879046 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.879251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.879233 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.880940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.880918 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.881114 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.881098 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.899264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.899239 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfnb\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.899516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.899499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:55.999594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:55.999571 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:56.323907 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:43:56.323873 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21adecb0_3bec_4eda_abcb_88f0d049577c.slice/crio-ba41447b588a57b6b480d060d3296a819f3842ebfa1b5a512f70a3e1d98dbb40 WatchSource:0}: Error finding container ba41447b588a57b6b480d060d3296a819f3842ebfa1b5a512f70a3e1d98dbb40: Status 404 returned error can't find the container with id ba41447b588a57b6b480d060d3296a819f3842ebfa1b5a512f70a3e1d98dbb40 Oct 29 22:43:56.326414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.326392 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj"] Oct 29 22:43:56.327035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.326990 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:56.327093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.327068 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:56.327126 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.327099 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:56.460498 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.460474 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:43:56.489741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.489716 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.489854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.489764 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.489854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.489827 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.489971 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490012 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490138 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490126 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490133 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.490299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490246 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490303 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490443 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.490863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490403 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket\") pod \"60d986f4-1e69-416d-89a0-79f70a431e83\" (UID: \"60d986f4-1e69-416d-89a0-79f70a431e83\") " Oct 29 22:43:56.490863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490808 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.490863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490821 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:56.490863 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.490825 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.491696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.491463 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d986f4-1e69-416d-89a0-79f70a431e83-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.491696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.491485 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.491696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.491500 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.491696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.491512 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.491696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.491529 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.492374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.492273 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:56.492490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.492470 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.492857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.492837 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "kube-api-access-rr6m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:56.493070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.493055 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token" pod "60d986f4-1e69-416d-89a0-79f70a431e83" (UID: "60d986f4-1e69-416d-89a0-79f70a431e83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:56.567253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.563219 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l"] Oct 29 22:43:56.567253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.564428 5073 cpu_manager.go:414] "containerMap: removing container" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" Oct 29 22:43:56.567253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.564450 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" Oct 29 22:43:56.567253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.564642 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" containerName="istio-proxy" Oct 29 22:43:56.576038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.575867 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l"] Oct 29 22:43:56.576038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.575956 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.577671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.577644 5073 generic.go:356] "Generic (PLEG): container finished" podID="60d986f4-1e69-416d-89a0-79f70a431e83" containerID="f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d" exitCode=0 Oct 29 22:43:56.577919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.577876 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" Oct 29 22:43:56.578411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.578389 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" event={"ID":"60d986f4-1e69-416d-89a0-79f70a431e83","Type":"ContainerDied","Data":"f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d"} Oct 29 22:43:56.578547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.578534 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt" event={"ID":"60d986f4-1e69-416d-89a0-79f70a431e83","Type":"ContainerDied","Data":"c8da38ea7e810a95ffe73ac8d74cf079125f5ee3810c94ce96ed6d165e366000"} Oct 29 22:43:56.578654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.578643 5073 scope.go:122] "RemoveContainer" containerID="f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d" Oct 29 22:43:56.580894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.580875 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw" Oct 29 22:43:56.591243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.589797 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" event={"ID":"21adecb0-3bec-4eda-abcb-88f0d049577c","Type":"ContainerStarted","Data":"51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467"} Oct 29 22:43:56.591243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.589844 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" event={"ID":"21adecb0-3bec-4eda-abcb-88f0d049577c","Type":"ContainerStarted","Data":"ba41447b588a57b6b480d060d3296a819f3842ebfa1b5a512f70a3e1d98dbb40"} Oct 29 22:43:56.592580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592298 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592336 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtf9\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592430 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592521 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592621 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592749 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592775 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592816 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.592849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592838 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.593229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592896 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d986f4-1e69-416d-89a0-79f70a431e83-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.593229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592906 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr6m2\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-kube-api-access-rr6m2\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.593229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592915 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d986f4-1e69-416d-89a0-79f70a431e83-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.593229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.592924 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d986f4-1e69-416d-89a0-79f70a431e83-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.598536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.598517 5073 scope.go:122] "RemoveContainer" containerID="f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d" Oct 29 22:43:56.599906 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:43:56.599880 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d\": container with ID starting with f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d not found: ID does not exist" containerID="f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d" Oct 29 22:43:56.600000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.599918 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d"} err="failed to get container status \"f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d\": rpc error: code = NotFound desc = could not find container \"f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d\": container with ID starting with f67d11a8c44f8c6f57caf4a74d30b776ac6d422ae4039366d2d372fc268e730d not found: ID does not exist" Oct 29 22:43:56.686461 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.686403 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podStartSLOduration=17.686385317 podStartE2EDuration="17.686385317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:56.684037707 +0000 UTC m=+1751.999431784" watchObservedRunningTime="2026-10-29 22:43:56.686385317 +0000 UTC m=+1752.001779398" Oct 29 22:43:56.693923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.693871 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694036 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694077 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694145 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694304 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694342 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694527 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694201 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.694990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.694975 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.695056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.695005 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtf9\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.695112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.695082 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.695218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.695201 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.695537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.695514 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.695981 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.695956 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.697717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.697693 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.697874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.697842 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.719086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.719060 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.721765 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.721731 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtf9\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.729396 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.729377 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt"] Oct 29 22:43:56.735641 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.735619 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-5dhwt"] Oct 29 22:43:56.777082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.777057 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw"] Oct 29 22:43:56.782412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.782385 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-pvvrw"] Oct 29 22:43:56.840940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.840870 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7fbd69-c165-41e7-a768-2443557073ff" path="/var/lib/kubelet/pods/0b7fbd69-c165-41e7-a768-2443557073ff/volumes" Oct 29 22:43:56.841495 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.841474 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d986f4-1e69-416d-89a0-79f70a431e83" path="/var/lib/kubelet/pods/60d986f4-1e69-416d-89a0-79f70a431e83/volumes" Oct 29 22:43:56.921580 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.921538 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:56.999954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:56.999757 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:43:57.508205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:57.495601 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l"] Oct 29 22:43:57.522174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:57.518945 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:57.522470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:57.522435 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:57.522652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:57.522624 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:43:57.672092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:57.672051 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" event={"ID":"303790fa-f5a1-46e8-a020-6520f938da55","Type":"ContainerStarted","Data":"5625ec8f0825f84208aefbc58484c0de73ff2d19c2982c5a225cbe21f6e88a74"} Oct 29 22:43:58.002216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.002116 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:43:58.002216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.002190 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:43:58.704998 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.704961 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" event={"ID":"303790fa-f5a1-46e8-a020-6520f938da55","Type":"ContainerStarted","Data":"98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d"} Oct 29 22:43:58.802739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.802682 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podStartSLOduration=19.802662665 podStartE2EDuration="19.802662665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:58.78395114 +0000 UTC m=+1754.099345217" watchObservedRunningTime="2026-10-29 22:43:58.802662665 +0000 UTC m=+1754.118056746" Oct 29 22:43:58.960732 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.960694 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:43:58.962986 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.962959 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:43:58.963816 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:58.963228 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 29 22:43:59.000270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:59.000242 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:43:59.000413 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:59.000308 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:43:59.957174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:59.956470 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:43:59.957174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:59.956526 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 29 22:44:00.000005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:43:59.999973 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.000130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:00.000040 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:44:00.931173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:00.930591 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.931173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:00.930669 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 29 22:44:01.000694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:01.000663 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.001185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:01.000726 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:44:01.960255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:01.947109 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.960255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:01.947187 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 29 22:44:01.999915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:01.999874 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.000087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:01.999936 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:44:02.942885 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:02.942845 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.968291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:02.968247 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 29 22:44:03.001454 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:03.001415 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:44:03.001725 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:03.001695 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:44:03.933726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:03.933677 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 29 22:44:03.933924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:03.933742 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 29 22:44:04.004959 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:04.004933 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:44:04.005425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:04.005406 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:44:04.005985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:04.005963 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:44:04.929919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:04.929847 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:44:04.930408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:04.930377 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:44:04.931240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:04.931219 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:44:05.746084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:05.746050 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs"] Oct 29 22:44:05.746587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:05.746311 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" containerID="cri-o://ca401e12bd329f5a027088c850e0e32cefc8361500759de4e37ea3e1a6e2d5a3" gracePeriod=30 Oct 29 22:44:09.256797 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:44:09.256757 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:44:10.657309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:10.657274 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt"] Oct 29 22:44:10.671839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:10.671811 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-433712-x8hgt"] Oct 29 22:44:10.842849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:10.842815 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6c99d4-38b9-474a-a852-f14057e908eb" path="/var/lib/kubelet/pods/fa6c99d4-38b9-474a-a852-f14057e908eb/volumes" Oct 29 22:44:10.956191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:10.956044 5073 generic.go:356] "Generic (PLEG): container finished" podID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerID="ca401e12bd329f5a027088c850e0e32cefc8361500759de4e37ea3e1a6e2d5a3" exitCode=0 Oct 29 22:44:10.956191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:10.956100 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" event={"ID":"37686afe-62ab-4be1-80d0-ec40cad5ece7","Type":"ContainerDied","Data":"ca401e12bd329f5a027088c850e0e32cefc8361500759de4e37ea3e1a6e2d5a3"} Oct 29 22:44:11.063745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.063694 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4"] Oct 29 22:44:11.115730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.102292 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c06e05-z76k4"] Oct 29 22:44:11.507374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.505642 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.583601 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.583811 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.583879 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.583928 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.584057 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.584129 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.584184 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.584220 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.584250 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket\") pod \"37686afe-62ab-4be1-80d0-ec40cad5ece7\" (UID: \"37686afe-62ab-4be1-80d0-ec40cad5ece7\") " Oct 29 22:44:11.590175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.587714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:11.609250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.591244 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.618244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.613284 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:11.618244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.613358 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:11.618478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.618438 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:11.630179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.618598 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:11.695199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.691427 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:11.695199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.691485 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "kube-api-access-xhqns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:11.695199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.691560 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:11.695199 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.691622 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token" pod "37686afe-62ab-4be1-80d0-ec40cad5ece7" (UID: "37686afe-62ab-4be1-80d0-ec40cad5ece7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695895 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695926 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqns\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-kube-api-access-xhqns\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695940 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37686afe-62ab-4be1-80d0-ec40cad5ece7-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695954 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695966 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695979 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.695990 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.705179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.696001 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37686afe-62ab-4be1-80d0-ec40cad5ece7-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.993281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.993238 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" event={"ID":"37686afe-62ab-4be1-80d0-ec40cad5ece7","Type":"ContainerDied","Data":"da2c99b60a0fd5426e95fb38d6b170ef5f30c6e05b037aa9f7c79f0da438ddcb"} Oct 29 22:44:11.993449 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.993321 5073 scope.go:122] "RemoveContainer" containerID="ca401e12bd329f5a027088c850e0e32cefc8361500759de4e37ea3e1a6e2d5a3" Oct 29 22:44:11.993525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:11.993497 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs" Oct 29 22:44:12.086422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:12.086341 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs"] Oct 29 22:44:12.096931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:12.096904 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-q9drs"] Oct 29 22:44:12.840739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:12.840711 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" path="/var/lib/kubelet/pods/37686afe-62ab-4be1-80d0-ec40cad5ece7/volumes" Oct 29 22:44:12.841235 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:12.841219 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5998404b-22e0-4ea9-b8dc-fd940e9337fe" path="/var/lib/kubelet/pods/5998404b-22e0-4ea9-b8dc-fd940e9337fe/volumes" Oct 29 22:44:17.375001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:17.374956 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z"] Oct 29 22:44:18.956173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.955403 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp"] Oct 29 22:44:18.956688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.956590 5073 cpu_manager.go:414] "containerMap: removing container" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" Oct 29 22:44:18.956688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.956611 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" Oct 29 22:44:18.961231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.956853 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="37686afe-62ab-4be1-80d0-ec40cad5ece7" containerName="istio-proxy" Oct 29 22:44:18.967486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.967459 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp"] Oct 29 22:44:18.969476 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.969454 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:18.995322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:18.989484 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-63db1a-openshift-default-dockercfg-x2525\"" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059289 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059379 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059440 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059662 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059711 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059760 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059887 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.080940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.059981 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.120675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.104219 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc"] Oct 29 22:44:19.120675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.104479 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" containerID="cri-o://6f6b5bd8ac0ab23bd5ff7f54f419fe5f2baefb61d0c43581718cd1d776125f19" gracePeriod=30 Oct 29 22:44:19.152002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.136961 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47"] Oct 29 22:44:19.154138 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.152317 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" containerID="cri-o://9ebda31ffd63d8bfe696d163459acee1b872ab449d528f929e60a117bd7ce1a0" gracePeriod=30 Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.165001 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166323 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166412 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166461 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166509 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166621 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166703 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166773 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.166837 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.167958 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.168276 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.168520 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.169118 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.178389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.169631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.191653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.189380 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.216456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.216347 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.220818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.220760 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.308036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.307929 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.457250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.457101 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:19.800112 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:44:19.800082 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba90d129_8ae5_4a2d_954a_cc532543b6ac.slice/crio-e522b7e97523e129ed6d8685a3ff02eb4c27eb8cba80022d7d65c7bf2bec5619 WatchSource:0}: Error finding container e522b7e97523e129ed6d8685a3ff02eb4c27eb8cba80022d7d65c7bf2bec5619: Status 404 returned error can't find the container with id e522b7e97523e129ed6d8685a3ff02eb4c27eb8cba80022d7d65c7bf2bec5619 Oct 29 22:44:19.801216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.801190 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp"] Oct 29 22:44:19.802909 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.802876 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:19.802983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.802944 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:19.802983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:19.802975 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:20.380859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.380810 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" event={"ID":"ba90d129-8ae5-4a2d-954a-cc532543b6ac","Type":"ContainerStarted","Data":"85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be"} Oct 29 22:44:20.381282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.380879 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" event={"ID":"ba90d129-8ae5-4a2d-954a-cc532543b6ac","Type":"ContainerStarted","Data":"e522b7e97523e129ed6d8685a3ff02eb4c27eb8cba80022d7d65c7bf2bec5619"} Oct 29 22:44:20.464178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.460590 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:20.464178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.460775 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:20.464178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.460847 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:20.477239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.467524 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podStartSLOduration=2.467506039 podStartE2EDuration="2.467506039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:20.46056063 +0000 UTC m=+1775.775954710" watchObservedRunningTime="2026-10-29 22:44:20.467506039 +0000 UTC m=+1775.782900118" Oct 29 22:44:20.603177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.590319 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl"] Oct 29 22:44:20.603177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.590547 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" containerID="cri-o://e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651" gracePeriod=30 Oct 29 22:44:20.891329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.890043 5073 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:44:20.891329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:20.890130 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:44:21.104195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:21.103970 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6"] Oct 29 22:44:21.104195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:21.104181 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" containerID="cri-o://ecc75e5b1caaa78d73d8c23162042b51d0921bf9c0739b1a4096a04a88e5320f" gracePeriod=30 Oct 29 22:44:21.458255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:21.458223 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:21.458647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:21.458287 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:22.457729 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:22.457697 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:22.457931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:22.457759 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:23.457941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:23.457881 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:23.458463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:23.457958 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:24.462172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.460951 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:24.462172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.461027 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:24.474623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.474348 5073 generic.go:356] "Generic (PLEG): container finished" podID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerID="6f6b5bd8ac0ab23bd5ff7f54f419fe5f2baefb61d0c43581718cd1d776125f19" exitCode=0 Oct 29 22:44:24.474623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.474539 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" event={"ID":"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4","Type":"ContainerDied","Data":"6f6b5bd8ac0ab23bd5ff7f54f419fe5f2baefb61d0c43581718cd1d776125f19"} Oct 29 22:44:24.477022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.476991 5073 generic.go:356] "Generic (PLEG): container finished" podID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerID="9ebda31ffd63d8bfe696d163459acee1b872ab449d528f929e60a117bd7ce1a0" exitCode=0 Oct 29 22:44:24.477180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.477049 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" event={"ID":"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f","Type":"ContainerDied","Data":"9ebda31ffd63d8bfe696d163459acee1b872ab449d528f929e60a117bd7ce1a0"} Oct 29 22:44:24.699253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.699221 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:44:24.796613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.796576 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:44:24.898867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.898839 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.899106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899082 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.899250 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899231 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899534 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899673 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899752 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899806 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899857 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899883 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899950 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.899978 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900043 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900070 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900093 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900210 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900317 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900365 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900452 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy\") pod \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\" (UID: \"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900602 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.900955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.900638 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert\") pod \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\" (UID: \"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f\") " Oct 29 22:44:24.901988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.901835 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.901988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.901858 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.901988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.901880 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:24.903782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.903557 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:24.903782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.903615 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.903943 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.903868 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.904210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.904047 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.904599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.904517 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:24.904906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.904879 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.905273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.905024 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.905273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.905170 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.907703 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.907666 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:24.908228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.908200 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:24.933358 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.933319 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "kube-api-access-zgnfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.951562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.951514 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "kube-api-access-kw5x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.951879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.951841 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo" pod "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" (UID: "40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:24.952050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.952019 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:24.953289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:24.953254 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy" pod "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" (UID: "cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:25.004434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004405 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004433 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004442 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004451 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgnfh\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-kube-api-access-zgnfh\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004460 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004468 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004475 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004482 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004488 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004495 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw5x6\" (UniqueName: \"kubernetes.io/projected/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-kube-api-access-kw5x6\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004503 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004510 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004518 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004525 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.004599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.004534 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:25.457594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.457562 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:25.457736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.457629 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:25.486774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.486745 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" Oct 29 22:44:25.487286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.486766 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47" event={"ID":"40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f","Type":"ContainerDied","Data":"78b7d210eb1be036e858cb5dad79e85254c6e868832ba02fc8cf400ffe65a378"} Oct 29 22:44:25.487286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.486861 5073 scope.go:122] "RemoveContainer" containerID="9ebda31ffd63d8bfe696d163459acee1b872ab449d528f929e60a117bd7ce1a0" Oct 29 22:44:25.489019 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.488988 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" event={"ID":"cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4","Type":"ContainerDied","Data":"339b13b724b7beb08e47f819ebaf0bc5d356923f477abc560de8168879df9b18"} Oct 29 22:44:25.489122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.488997 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc" Oct 29 22:44:25.498844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.498816 5073 scope.go:122] "RemoveContainer" containerID="6f6b5bd8ac0ab23bd5ff7f54f419fe5f2baefb61d0c43581718cd1d776125f19" Oct 29 22:44:25.561054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.560980 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc"] Oct 29 22:44:25.574178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.573506 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-5b5qc"] Oct 29 22:44:25.641620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.641579 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:44:25.641787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.641651 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:44:25.642262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.642238 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47"] Oct 29 22:44:25.646075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:25.646047 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-dsl47"] Oct 29 22:44:26.099105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.099065 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172548 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172598 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172675 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172712 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172829 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172872 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172903 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.172973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.172943 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.175303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.175271 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token\") pod \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\" (UID: \"5223d9bb-d03a-473c-9d24-876c69f2dbe3\") " Oct 29 22:44:26.175651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.175624 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.176341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.176320 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.176599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.176580 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.176701 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.176676 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.176922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.176900 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.176997 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.176936 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:26.178432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.178395 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:26.179233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.179207 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:26.182650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.182619 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "kube-api-access-knvzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:26.183288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.183253 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo" pod "5223d9bb-d03a-473c-9d24-876c69f2dbe3" (UID: "5223d9bb-d03a-473c-9d24-876c69f2dbe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:26.278003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.277971 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278002 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278010 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knvzv\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-kube-api-access-knvzv\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278019 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278028 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278035 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278043 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.278253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.278050 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5223d9bb-d03a-473c-9d24-876c69f2dbe3-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:26.457543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.457510 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:44:26.457804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.457775 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:44:26.504709 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.504667 5073 generic.go:356] "Generic (PLEG): container finished" podID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerID="e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651" exitCode=0 Oct 29 22:44:26.505544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.504905 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" Oct 29 22:44:26.509832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.504926 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" event={"ID":"5223d9bb-d03a-473c-9d24-876c69f2dbe3","Type":"ContainerDied","Data":"e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651"} Oct 29 22:44:26.509832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.509402 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl" event={"ID":"5223d9bb-d03a-473c-9d24-876c69f2dbe3","Type":"ContainerDied","Data":"500e356f4cab9227fc52e9defb120e631235bc0acb5eaddbd58631aa9498cb8d"} Oct 29 22:44:26.509832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.509446 5073 scope.go:122] "RemoveContainer" containerID="e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651" Oct 29 22:44:26.517660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.515374 5073 generic.go:356] "Generic (PLEG): container finished" podID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerID="ecc75e5b1caaa78d73d8c23162042b51d0921bf9c0739b1a4096a04a88e5320f" exitCode=0 Oct 29 22:44:26.517660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.515508 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" event={"ID":"55317bea-e9ea-4d29-ad8f-4996674bd8fe","Type":"ContainerDied","Data":"ecc75e5b1caaa78d73d8c23162042b51d0921bf9c0739b1a4096a04a88e5320f"} Oct 29 22:44:26.529241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.526532 5073 scope.go:122] "RemoveContainer" containerID="e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651" Oct 29 22:44:26.529241 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:44:26.527883 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651\": container with ID starting with e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651 not found: ID does not exist" containerID="e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651" Oct 29 22:44:26.529241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.527917 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651"} err="failed to get container status \"e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651\": rpc error: code = NotFound desc = could not find container \"e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651\": container with ID starting with e479245029ec80c7a1697ab07db43ac78c11e6511568e0d4c52b08104330d651 not found: ID does not exist" Oct 29 22:44:26.573776 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.573745 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl"] Oct 29 22:44:26.586538 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.586503 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-86fhl"] Oct 29 22:44:26.943174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:26.942338 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:44:27.008210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.007073 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.014858 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015033 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015126 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015219 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015286 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015337 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015465 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.015530 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket\") pod \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\" (UID: \"55317bea-e9ea-4d29-ad8f-4996674bd8fe\") " Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.019925 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:27.022223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.021447 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:27.022883 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.022630 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:27.024749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.024369 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:27.024749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.024488 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:27.033117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.029623 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" path="/var/lib/kubelet/pods/40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f/volumes" Oct 29 22:44:27.033117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.031815 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" path="/var/lib/kubelet/pods/5223d9bb-d03a-473c-9d24-876c69f2dbe3/volumes" Oct 29 22:44:27.033117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.032370 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" path="/var/lib/kubelet/pods/cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4/volumes" Oct 29 22:44:27.033117 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.032839 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:27.035658 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.034882 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:27.053203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.041349 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "kube-api-access-njd6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:27.056275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.055359 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token" pod "55317bea-e9ea-4d29-ad8f-4996674bd8fe" (UID: "55317bea-e9ea-4d29-ad8f-4996674bd8fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120224 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120257 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120271 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njd6f\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-kube-api-access-njd6f\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120285 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120297 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120310 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120321 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120332 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55317bea-e9ea-4d29-ad8f-4996674bd8fe-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.122098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.120343 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55317bea-e9ea-4d29-ad8f-4996674bd8fe-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:27.464217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.464185 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:27.464777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.464752 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:27.465519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.465497 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:44:27.545201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.542760 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" Oct 29 22:44:27.545201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.542732 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6" event={"ID":"55317bea-e9ea-4d29-ad8f-4996674bd8fe","Type":"ContainerDied","Data":"496ec90c62effe12858c2d4414b2dcf4089ee5612de7ece4b3cf72e85ca0dc71"} Oct 29 22:44:27.545201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.542916 5073 scope.go:122] "RemoveContainer" containerID="ecc75e5b1caaa78d73d8c23162042b51d0921bf9c0739b1a4096a04a88e5320f" Oct 29 22:44:27.683172 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.677467 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6"] Oct 29 22:44:27.721226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:27.720662 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6hsq6"] Oct 29 22:44:28.843278 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:28.843245 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" path="/var/lib/kubelet/pods/55317bea-e9ea-4d29-ad8f-4996674bd8fe/volumes" Oct 29 22:44:32.398288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.398246 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" podUID="d246285c-98f5-4d1b-a35f-e279c5958921" containerName="main" containerID="cri-o://fd6628d0b5b428199e1f0ac2a2775e4f08f9c59e7bf82d05db5af6857d7e9ad5" gracePeriod=45 Oct 29 22:44:32.670835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.670790 5073 generic.go:356] "Generic (PLEG): container finished" podID="d246285c-98f5-4d1b-a35f-e279c5958921" containerID="fd6628d0b5b428199e1f0ac2a2775e4f08f9c59e7bf82d05db5af6857d7e9ad5" exitCode=0 Oct 29 22:44:32.671164 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.670904 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" event={"ID":"d246285c-98f5-4d1b-a35f-e279c5958921","Type":"ContainerDied","Data":"fd6628d0b5b428199e1f0ac2a2775e4f08f9c59e7bf82d05db5af6857d7e9ad5"} Oct 29 22:44:32.854792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.854091 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:44:32.957557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.955734 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home\") pod \"d246285c-98f5-4d1b-a35f-e279c5958921\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " Oct 29 22:44:32.957557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.955816 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs\" (UniqueName: \"kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs\") pod \"d246285c-98f5-4d1b-a35f-e279c5958921\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " Oct 29 22:44:32.957557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.955866 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b\" (UniqueName: \"kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b\") pod \"d246285c-98f5-4d1b-a35f-e279c5958921\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " Oct 29 22:44:32.957557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.956072 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm\") pod \"d246285c-98f5-4d1b-a35f-e279c5958921\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " Oct 29 22:44:32.957557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.956134 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir\") pod \"d246285c-98f5-4d1b-a35f-e279c5958921\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " Oct 29 22:44:32.957557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.956243 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache\") pod \"d246285c-98f5-4d1b-a35f-e279c5958921\" (UID: \"d246285c-98f5-4d1b-a35f-e279c5958921\") " Oct 29 22:44:32.957996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.957675 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home" pod "d246285c-98f5-4d1b-a35f-e279c5958921" (UID: "d246285c-98f5-4d1b-a35f-e279c5958921"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.961576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.960072 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir" pod "d246285c-98f5-4d1b-a35f-e279c5958921" (UID: "d246285c-98f5-4d1b-a35f-e279c5958921"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.961576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.960287 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache" pod "d246285c-98f5-4d1b-a35f-e279c5958921" (UID: "d246285c-98f5-4d1b-a35f-e279c5958921"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.961576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.960735 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs" pod "d246285c-98f5-4d1b-a35f-e279c5958921" (UID: "d246285c-98f5-4d1b-a35f-e279c5958921"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:32.963178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.961982 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm" pod "d246285c-98f5-4d1b-a35f-e279c5958921" (UID: "d246285c-98f5-4d1b-a35f-e279c5958921"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.969845 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:32.969801 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b" pod "d246285c-98f5-4d1b-a35f-e279c5958921" (UID: "d246285c-98f5-4d1b-a35f-e279c5958921"). InnerVolumeSpecName "kube-api-access-l2q2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:33.059225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.059189 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:33.059225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.059229 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:33.059474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.059244 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:33.059474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.059257 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246285c-98f5-4d1b-a35f-e279c5958921-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:33.059474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.059268 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246285c-98f5-4d1b-a35f-e279c5958921-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:33.059474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.059281 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2q2b\" (UniqueName: \"kubernetes.io/projected/d246285c-98f5-4d1b-a35f-e279c5958921-kube-api-access-l2q2b\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:33.694176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.682982 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" Oct 29 22:44:33.694176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.683047 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z" event={"ID":"d246285c-98f5-4d1b-a35f-e279c5958921","Type":"ContainerDied","Data":"f1280b2102094f409f0878a2d53350ffcfd05bd28137136fe8a958416e54babb"} Oct 29 22:44:33.694176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.683105 5073 scope.go:122] "RemoveContainer" containerID="fd6628d0b5b428199e1f0ac2a2775e4f08f9c59e7bf82d05db5af6857d7e9ad5" Oct 29 22:44:33.744028 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.743994 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z"] Oct 29 22:44:33.757933 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.757889 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d478b36e/e2e-lifecycle-model-d478b36e-kserve-75d69c6765-mz58z"] Oct 29 22:44:33.891790 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.891754 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4"] Oct 29 22:44:33.892427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892411 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" Oct 29 22:44:33.892427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892427 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892438 5073 cpu_manager.go:414] "containerMap: removing container" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892443 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892460 5073 cpu_manager.go:414] "containerMap: removing container" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892465 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892496 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892500 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892513 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d246285c-98f5-4d1b-a35f-e279c5958921" containerName="main" Oct 29 22:44:33.892529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892518 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d246285c-98f5-4d1b-a35f-e279c5958921" containerName="main" Oct 29 22:44:33.892760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892604 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5223d9bb-d03a-473c-9d24-876c69f2dbe3" containerName="istio-proxy" Oct 29 22:44:33.892760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892615 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d246285c-98f5-4d1b-a35f-e279c5958921" containerName="main" Oct 29 22:44:33.892760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892622 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb856c23-72ff-4d6e-8e2b-6255e5c8b0e4" containerName="istio-proxy" Oct 29 22:44:33.892760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892628 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a5e002-8dc8-47c4-a9fb-6c54fa5a8d8f" containerName="istio-proxy" Oct 29 22:44:33.892760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.892635 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="55317bea-e9ea-4d29-ad8f-4996674bd8fe" containerName="istio-proxy" Oct 29 22:44:33.897263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.897241 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.918758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.918429 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg"] Oct 29 22:44:33.933321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.933298 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:33.956375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.949508 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4"] Oct 29 22:44:33.956375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.952859 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg"] Oct 29 22:44:33.974610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974587 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.974761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974631 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.974761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974718 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.974860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974759 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.974860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974801 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.974964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974937 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.975034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.974980 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.975486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.975464 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:33.975696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:33.975517 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfh4\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.076855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.076812 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.076916 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.076953 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.076984 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077014 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077059 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077097 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077132 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077188 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077338 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077415 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077472 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077642 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077496 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077642 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077506 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077642 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077595 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077692 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077728 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077775 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nd6q\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077810 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.077963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077857 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077875 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077923 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.077963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.077949 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfh4\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.080022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.079993 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.080700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.080665 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.099295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.099271 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.099630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.099611 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfh4\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.179643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179612 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.179786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179646 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.179786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179692 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.179786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179721 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.179912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179814 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.179912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179879 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nd6q\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.179911 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180017 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180025 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180064 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180256 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180136 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180262 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180299 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.180815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.180796 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.182429 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.182404 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.182801 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.182781 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.200461 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.200438 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.200677 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.200655 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nd6q\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.233210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.222368 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:34.261707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.261673 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:34.556340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.556310 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4"] Oct 29 22:44:34.556624 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:44:34.556603 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6247cc5b_7cda_452f_bd91_9e22ec75955a.slice/crio-df1eaa0bd0b88eb273544ee1f351c9473d4fb753a57361e1c9b40744c5055db2 WatchSource:0}: Error finding container df1eaa0bd0b88eb273544ee1f351c9473d4fb753a57361e1c9b40744c5055db2: Status 404 returned error can't find the container with id df1eaa0bd0b88eb273544ee1f351c9473d4fb753a57361e1c9b40744c5055db2 Oct 29 22:44:34.559629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.559591 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:34.559736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.559672 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:34.559736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.559712 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:34.635573 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.632188 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg"] Oct 29 22:44:34.639907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.638962 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:34.639907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.639052 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:34.639907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.639089 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:34.697396 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.697361 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" event={"ID":"6247cc5b-7cda-452f-bd91-9e22ec75955a","Type":"ContainerStarted","Data":"d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b"} Oct 29 22:44:34.697787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.697425 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" event={"ID":"6247cc5b-7cda-452f-bd91-9e22ec75955a","Type":"ContainerStarted","Data":"df1eaa0bd0b88eb273544ee1f351c9473d4fb753a57361e1c9b40744c5055db2"} Oct 29 22:44:34.700183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.700137 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" event={"ID":"b1f39109-3fc4-415b-aa44-bade3be9f985","Type":"ContainerStarted","Data":"5d9874ceab7ac7d0fbdd2f5d7d63da3caaeda4b35e2dabc1455f5cb22e6e5d10"} Oct 29 22:44:34.780088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.778187 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podStartSLOduration=1.7781734280000001 podStartE2EDuration="1.778173428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:34.775385538 +0000 UTC m=+1790.090779618" watchObservedRunningTime="2026-10-29 22:44:34.778173428 +0000 UTC m=+1790.093567502" Oct 29 22:44:34.855692 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:34.855608 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d246285c-98f5-4d1b-a35f-e279c5958921" path="/var/lib/kubelet/pods/d246285c-98f5-4d1b-a35f-e279c5958921/volumes" Oct 29 22:44:35.228176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:35.227520 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:35.231180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:35.229786 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:35.231180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:35.229841 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:35.739175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:35.729881 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" event={"ID":"b1f39109-3fc4-415b-aa44-bade3be9f985","Type":"ContainerStarted","Data":"f5816bb7cc48f9aa8425bad733ab23ffa828945da140238f95c54cf743f09891"} Oct 29 22:44:35.958200 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:35.948972 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podStartSLOduration=2.94894912 podStartE2EDuration="2.94894912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:35.81362525 +0000 UTC m=+1791.129019353" watchObservedRunningTime="2026-10-29 22:44:35.94894912 +0000 UTC m=+1791.264343200" Oct 29 22:44:36.229179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:36.227605 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:36.229179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:36.227667 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:36.262636 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:36.262581 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:36.264087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:36.263997 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:44:36.264087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:36.264038 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 22:44:37.229033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:37.228567 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:37.229033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:37.228630 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:37.265068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:37.264948 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:44:37.265068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:37.265006 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 22:44:38.288116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:38.288085 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:38.288767 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:38.288721 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:38.289338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:38.289299 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:44:38.289493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:38.289475 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 22:44:39.246175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:39.240192 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:39.246175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:39.240263 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:39.263302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:39.263131 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:44:39.263302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:39.263245 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 22:44:39.289363 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:44:39.289330 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:44:40.227546 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:40.227506 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:40.227837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:40.227806 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:40.263923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:40.263890 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:44:40.264222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:40.264180 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 22:44:41.227800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:41.227378 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:44:41.227800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:41.227442 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:44:41.262799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:41.262752 5073 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:44:41.263002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:41.262823 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 22:44:42.233217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:42.233138 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:42.233758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:42.233621 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:42.234604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:42.234583 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:44:42.269947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:42.269906 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:42.270332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:42.270302 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:42.271161 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:42.271113 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:44:43.958131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:43.958091 5073 generic.go:356] "Generic (PLEG): container finished" podID="790bb1ce-002a-4745-b825-467d8872aaec" containerID="a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f" exitCode=0 Oct 29 22:44:43.963252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:43.959501 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" event={"ID":"790bb1ce-002a-4745-b825-467d8872aaec","Type":"ContainerDied","Data":"a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f"} Oct 29 22:44:43.963252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:43.959985 5073 scope.go:122] "RemoveContainer" containerID="a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f" Oct 29 22:44:43.963252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:43.962738 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:43.963252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:43.962819 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:43.963252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:43.962867 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:44:44.968729 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:44.968689 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" event={"ID":"790bb1ce-002a-4745-b825-467d8872aaec","Type":"ContainerStarted","Data":"3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05"} Oct 29 22:44:45.184180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.184019 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:44:45.195086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.195060 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:44:45.197533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.197511 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:44:45.248395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.248312 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:44:45.842196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.842163 5073 scope.go:122] "RemoveContainer" containerID="137ff07fcf9251bd0648c3f440399af83a57bceb9612cce43a498fe6209ebe45" Oct 29 22:44:45.936669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.936633 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:44:45.937896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.937866 5073 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:44:45.938036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.937927 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 29 22:44:45.957958 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:45.957933 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:44:46.932182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:46.920476 5073 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:44:46.932182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:46.920544 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 29 22:44:47.914472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:47.913481 5073 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:44:47.914472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:47.913552 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 29 22:44:48.743714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:48.743681 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb"] Oct 29 22:44:48.781455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:48.781420 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2"] Oct 29 22:44:48.940329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:48.940293 5073 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:44:48.940601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:48.940571 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 29 22:44:49.914993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:49.914526 5073 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:44:49.914993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:49.914584 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 29 22:44:50.919037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:50.919007 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:44:50.920065 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:50.920046 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:44:52.943173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:52.942297 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:44:53.745795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:53.745748 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" podUID="55cbb1be-56f4-4b03-84ba-c05d7d909594" containerName="payload-processing" containerID="cri-o://52fab272e7ae74cd742151b463f25cb36ccc48e61a2531d7478c262208ae7886" gracePeriod=25 Oct 29 22:44:53.781936 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:53.781885 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" podUID="a5576976-85ba-42c5-80f2-f2801ab2d9f9" containerName="payload-pre-processing" containerID="cri-o://8e21f927469684a036a8f306837aaf7ff4058dbd27fd508b1c9992a78276ed4a" gracePeriod=25 Oct 29 22:44:54.074403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.074314 5073 generic.go:356] "Generic (PLEG): container finished" podID="55cbb1be-56f4-4b03-84ba-c05d7d909594" containerID="52fab272e7ae74cd742151b463f25cb36ccc48e61a2531d7478c262208ae7886" exitCode=0 Oct 29 22:44:54.074878 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.074507 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" event={"ID":"55cbb1be-56f4-4b03-84ba-c05d7d909594","Type":"ContainerDied","Data":"52fab272e7ae74cd742151b463f25cb36ccc48e61a2531d7478c262208ae7886"} Oct 29 22:44:54.079345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.079034 5073 generic.go:356] "Generic (PLEG): container finished" podID="a5576976-85ba-42c5-80f2-f2801ab2d9f9" containerID="8e21f927469684a036a8f306837aaf7ff4058dbd27fd508b1c9992a78276ed4a" exitCode=0 Oct 29 22:44:54.080522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.080059 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" event={"ID":"a5576976-85ba-42c5-80f2-f2801ab2d9f9","Type":"ContainerDied","Data":"8e21f927469684a036a8f306837aaf7ff4058dbd27fd508b1c9992a78276ed4a"} Oct 29 22:44:54.141820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.141789 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:44:54.258173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.235477 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume\") pod \"55cbb1be-56f4-4b03-84ba-c05d7d909594\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " Oct 29 22:44:54.258173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.235551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb\" (UniqueName: \"kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb\") pod \"55cbb1be-56f4-4b03-84ba-c05d7d909594\" (UID: \"55cbb1be-56f4-4b03-84ba-c05d7d909594\") " Oct 29 22:44:54.258173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.239131 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume" pod "55cbb1be-56f4-4b03-84ba-c05d7d909594" (UID: "55cbb1be-56f4-4b03-84ba-c05d7d909594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:54.258173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.244731 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb" pod "55cbb1be-56f4-4b03-84ba-c05d7d909594" (UID: "55cbb1be-56f4-4b03-84ba-c05d7d909594"). InnerVolumeSpecName "kube-api-access-jpbcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:54.259387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.259302 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55cbb1be-56f4-4b03-84ba-c05d7d909594-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:54.259387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.259335 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbcb\" (UniqueName: \"kubernetes.io/projected/55cbb1be-56f4-4b03-84ba-c05d7d909594-kube-api-access-jpbcb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:54.385111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.385086 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:44:54.461798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.461762 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume\") pod \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " Oct 29 22:44:54.462218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.462196 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t\" (UniqueName: \"kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t\") pod \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\" (UID: \"a5576976-85ba-42c5-80f2-f2801ab2d9f9\") " Oct 29 22:44:54.462313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.462205 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume" pod "a5576976-85ba-42c5-80f2-f2801ab2d9f9" (UID: "a5576976-85ba-42c5-80f2-f2801ab2d9f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:54.462766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.462743 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576976-85ba-42c5-80f2-f2801ab2d9f9-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:54.464427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.464407 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t" pod "a5576976-85ba-42c5-80f2-f2801ab2d9f9" (UID: "a5576976-85ba-42c5-80f2-f2801ab2d9f9"). InnerVolumeSpecName "kube-api-access-95w7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:54.564423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:54.564393 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95w7t\" (UniqueName: \"kubernetes.io/projected/a5576976-85ba-42c5-80f2-f2801ab2d9f9-kube-api-access-95w7t\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.176841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.175115 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" Oct 29 22:44:55.177405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.177036 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2" event={"ID":"a5576976-85ba-42c5-80f2-f2801ab2d9f9","Type":"ContainerDied","Data":"e2ee90665a2fb770fa14ab06f96eff4f0fd2cfd29afb01aef59bc528c4f85129"} Oct 29 22:44:55.177405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.177108 5073 scope.go:122] "RemoveContainer" containerID="8e21f927469684a036a8f306837aaf7ff4058dbd27fd508b1c9992a78276ed4a" Oct 29 22:44:55.182415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.180400 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" event={"ID":"55cbb1be-56f4-4b03-84ba-c05d7d909594","Type":"ContainerDied","Data":"100ae690c692b12fa96d2407d9a3d8b9258ff02db040cd003ddb78f6c98a03b4"} Oct 29 22:44:55.182415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.180566 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb" Oct 29 22:44:55.205125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.205095 5073 scope.go:122] "RemoveContainer" containerID="52fab272e7ae74cd742151b463f25cb36ccc48e61a2531d7478c262208ae7886" Oct 29 22:44:55.239194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.239119 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb"] Oct 29 22:44:55.249575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.249546 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63db1a-55c46bd5fd-2hxtb"] Oct 29 22:44:55.281309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.281253 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2"] Oct 29 22:44:55.285086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:55.285062 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63db1a-674f496d67-mblf2"] Oct 29 22:44:56.841954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:56.841918 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55cbb1be-56f4-4b03-84ba-c05d7d909594" path="/var/lib/kubelet/pods/55cbb1be-56f4-4b03-84ba-c05d7d909594/volumes" Oct 29 22:44:56.842653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:44:56.842632 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5576976-85ba-42c5-80f2-f2801ab2d9f9" path="/var/lib/kubelet/pods/a5576976-85ba-42c5-80f2-f2801ab2d9f9/volumes" Oct 29 22:45:00.162214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.162179 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p"] Oct 29 22:45:00.162887 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.162871 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a5576976-85ba-42c5-80f2-f2801ab2d9f9" containerName="payload-pre-processing" Oct 29 22:45:00.162932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.162889 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5576976-85ba-42c5-80f2-f2801ab2d9f9" containerName="payload-pre-processing" Oct 29 22:45:00.162932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.162912 5073 cpu_manager.go:414] "containerMap: removing container" podUID="55cbb1be-56f4-4b03-84ba-c05d7d909594" containerName="payload-processing" Oct 29 22:45:00.162932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.162917 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55cbb1be-56f4-4b03-84ba-c05d7d909594" containerName="payload-processing" Oct 29 22:45:00.163034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.163027 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="55cbb1be-56f4-4b03-84ba-c05d7d909594" containerName="payload-processing" Oct 29 22:45:00.163068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.163045 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5576976-85ba-42c5-80f2-f2801ab2d9f9" containerName="payload-pre-processing" Oct 29 22:45:00.168280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.168257 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.192655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.192623 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn"] Oct 29 22:45:00.197661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.197637 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.221904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.217608 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p"] Oct 29 22:45:00.223705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.223675 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl"] Oct 29 22:45:00.229110 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.229079 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.232599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.232574 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh"] Oct 29 22:45:00.242880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.242854 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn"] Oct 29 22:45:00.243882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.243857 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.251266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.251238 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl"] Oct 29 22:45:00.254610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.254588 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh"] Oct 29 22:45:00.271597 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.271576 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd"] Oct 29 22:45:00.277168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.277128 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.277323 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.277307 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.277373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.277346 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrr8\" (UniqueName: \"kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.277539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.277524 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-bxr4p\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.277575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.277553 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgmq\" (UniqueName: \"kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq\") pod \"maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.277610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.277583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p85w\" (UniqueName: \"kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w\") pod \"maas-api-key-cleanup-29888565-bxr4p\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.281312 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.281292 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml"] Oct 29 22:45:00.281448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.281428 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.285922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.285904 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.295317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.295286 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:45:00.314703 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.314682 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml"] Oct 29 22:45:00.315529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.315508 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd"] Oct 29 22:45:00.324700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.321469 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m"] Oct 29 22:45:00.328370 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.327829 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.339743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.339717 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t"] Oct 29 22:45:00.351887 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.351856 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m"] Oct 29 22:45:00.352025 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.351982 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.379853 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.379828 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t"] Oct 29 22:45:00.381088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.380974 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.381233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381113 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.381233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381139 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvs7\" (UniqueName: \"kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.381233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381196 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.381577 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381548 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrr8\" (UniqueName: \"kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.381676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381633 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2scv\" (UniqueName: \"kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv\") pod \"maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.381740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381680 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjcb\" (UniqueName: \"kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb\") pod \"maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.381828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381810 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.381896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381877 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.381952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.381924 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-bxr4p\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.382699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.382676 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-bxr4p\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.383201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.383139 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgmq\" (UniqueName: \"kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq\") pod \"maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.383488 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.383468 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p85w\" (UniqueName: \"kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w\") pod \"maas-api-key-cleanup-29888565-bxr4p\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.383604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.383585 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkvm\" (UniqueName: \"kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.383661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.383620 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.383715 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.383651 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.384388 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.384365 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.404564 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.404536 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgmq\" (UniqueName: \"kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq\") pod \"maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.408441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.408417 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p85w\" (UniqueName: \"kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w\") pod \"maas-api-key-cleanup-29888565-bxr4p\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.423273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.408039 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrr8\" (UniqueName: \"kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8\") pod \"maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.484870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.484840 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.484985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.484886 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvs7\" (UniqueName: \"kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.484985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.484916 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.484985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.484960 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2scv\" (UniqueName: \"kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv\") pod \"maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.485175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.484996 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjcb\" (UniqueName: \"kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb\") pod \"maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.485175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485017 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:00.485175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485066 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvh2\" (UniqueName: \"kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.485368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485179 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.485368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485306 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.485471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485390 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.485471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485440 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkvm\" (UniqueName: \"kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.485683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485657 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.485752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485732 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.485839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485821 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.485950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.485935 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.507589 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.507562 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvs7\" (UniqueName: \"kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7\") pod \"maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.507947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.507925 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2scv\" (UniqueName: \"kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv\") pod \"maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.508042 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.507971 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjcb\" (UniqueName: \"kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb\") pod \"maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.508042 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.508011 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkvm\" (UniqueName: \"kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.535523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.535491 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:00.569196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.565507 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:00.587704 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.585054 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:00.590118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.590068 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvh2\" (UniqueName: \"kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.590341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.590313 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.591324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.591292 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.600752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.600727 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:00.615104 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.614505 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvh2\" (UniqueName: \"kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.619890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.619681 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:00.687263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.673571 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:00.687263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.686872 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:00.855536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:00.855088 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p"] Oct 29 22:45:01.145654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.145618 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml"] Oct 29 22:45:01.147651 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.147622 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1aed31_3ae5_409e_a08c_2f1804124a3b.slice/crio-5920529a9a5821cc6a0af23618aa3bd5413bb80fd699275781db5c3aa963b557 WatchSource:0}: Error finding container 5920529a9a5821cc6a0af23618aa3bd5413bb80fd699275781db5c3aa963b557: Status 404 returned error can't find the container with id 5920529a9a5821cc6a0af23618aa3bd5413bb80fd699275781db5c3aa963b557 Oct 29 22:45:01.162676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.162608 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn"] Oct 29 22:45:01.168641 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.168612 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ccf14b_4022_4e3a_9757_458e77b0c827.slice/crio-9104719cc62c29c5696861d7a4ce0352d5f1670197c5572e40b45bdf8c73b434 WatchSource:0}: Error finding container 9104719cc62c29c5696861d7a4ce0352d5f1670197c5572e40b45bdf8c73b434: Status 404 returned error can't find the container with id 9104719cc62c29c5696861d7a4ce0352d5f1670197c5572e40b45bdf8c73b434 Oct 29 22:45:01.176503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.176475 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t"] Oct 29 22:45:01.177194 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.177142 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d6c86d_53c7_47ce_9597_d84fac5f3ae8.slice/crio-02b7c0a076b597fc2d9fe676a200385dc5801437c5faa3c48c7120386c83b47e WatchSource:0}: Error finding container 02b7c0a076b597fc2d9fe676a200385dc5801437c5faa3c48c7120386c83b47e: Status 404 returned error can't find the container with id 02b7c0a076b597fc2d9fe676a200385dc5801437c5faa3c48c7120386c83b47e Oct 29 22:45:01.182875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.182847 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl"] Oct 29 22:45:01.187103 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.187068 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103d267a_055f_4afe_b5f7_63ea9dca3525.slice/crio-8bc7961c299f45f642a2e92b46f594e243fb5f6ba62ebab7a5c597d5d3d79a41 WatchSource:0}: Error finding container 8bc7961c299f45f642a2e92b46f594e243fb5f6ba62ebab7a5c597d5d3d79a41: Status 404 returned error can't find the container with id 8bc7961c299f45f642a2e92b46f594e243fb5f6ba62ebab7a5c597d5d3d79a41 Oct 29 22:45:01.199229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.199201 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh"] Oct 29 22:45:01.203871 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.203829 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594df621_447e_4232_ab84_67bad2b0b940.slice/crio-f17818bdb2efa7e900a4b416be5d700645bb7feee87114d886046a30b3591edf WatchSource:0}: Error finding container f17818bdb2efa7e900a4b416be5d700645bb7feee87114d886046a30b3591edf: Status 404 returned error can't find the container with id f17818bdb2efa7e900a4b416be5d700645bb7feee87114d886046a30b3591edf Oct 29 22:45:01.255365 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.255335 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerStarted","Data":"5920529a9a5821cc6a0af23618aa3bd5413bb80fd699275781db5c3aa963b557"} Oct 29 22:45:01.256657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.256636 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerStarted","Data":"02b7c0a076b597fc2d9fe676a200385dc5801437c5faa3c48c7120386c83b47e"} Oct 29 22:45:01.257999 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.257969 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerStarted","Data":"f17818bdb2efa7e900a4b416be5d700645bb7feee87114d886046a30b3591edf"} Oct 29 22:45:01.259261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.259238 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerStarted","Data":"9104719cc62c29c5696861d7a4ce0352d5f1670197c5572e40b45bdf8c73b434"} Oct 29 22:45:01.260549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.260527 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerStarted","Data":"9043377a0da34b56168f008a6ed5888d47e416043c249d69eb4a74e31088ce1a"} Oct 29 22:45:01.261755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.261731 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerStarted","Data":"8bc7961c299f45f642a2e92b46f594e243fb5f6ba62ebab7a5c597d5d3d79a41"} Oct 29 22:45:01.278393 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.278357 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8076aef0_abbb_4aba_9d5f_6afcabe62e28.slice/crio-c5792c25e3ced37a52b6506e5be0f50b8cb3875e1b1857b09cd2101db9afed50 WatchSource:0}: Error finding container c5792c25e3ced37a52b6506e5be0f50b8cb3875e1b1857b09cd2101db9afed50: Status 404 returned error can't find the container with id c5792c25e3ced37a52b6506e5be0f50b8cb3875e1b1857b09cd2101db9afed50 Oct 29 22:45:01.280309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.280288 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m"] Oct 29 22:45:01.280971 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:45:01.280938 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7029b8b_85ef_4cde_8e00_f592b71056a0.slice/crio-8cceaf9403a7cd0dc717808d22de93033b0f056f4e32e456409d556804f5ec40 WatchSource:0}: Error finding container 8cceaf9403a7cd0dc717808d22de93033b0f056f4e32e456409d556804f5ec40: Status 404 returned error can't find the container with id 8cceaf9403a7cd0dc717808d22de93033b0f056f4e32e456409d556804f5ec40 Oct 29 22:45:01.284463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:01.284438 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd"] Oct 29 22:45:02.272521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.272482 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerStarted","Data":"867a661cc857a6890e87a41f28916f9ca9f744dfe58dbf88f2e35f8de140e09c"} Oct 29 22:45:02.273007 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.272540 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerStarted","Data":"c5792c25e3ced37a52b6506e5be0f50b8cb3875e1b1857b09cd2101db9afed50"} Oct 29 22:45:02.274340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.274312 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerStarted","Data":"6c31110ade0478e2805928de78990c5043f71dacf22da5ec7a1124aced785d41"} Oct 29 22:45:02.275923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.275897 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerStarted","Data":"a640644309345b984e76580e2d9a0bb94e9842d4e19750e3daa49f54162f053c"} Oct 29 22:45:02.276070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.275945 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerStarted","Data":"8cceaf9403a7cd0dc717808d22de93033b0f056f4e32e456409d556804f5ec40"} Oct 29 22:45:02.277418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.277394 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerStarted","Data":"cc41784ae1941454aed42b2d016fa3464899bc91e1a220c9f1769d32ee16188e"} Oct 29 22:45:02.279085 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.279063 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerStarted","Data":"b75fec6a3d6abb4890be392fdc29513e961e52c268f37e21ee2ff011fc5824a8"} Oct 29 22:45:02.281001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.280975 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerStarted","Data":"f95e34909c0d3fff87053976c70c2163d84c62bf97d0ca959cf0805f4d110e5b"} Oct 29 22:45:02.282840 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.282813 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerStarted","Data":"b0491176be563cda4575f63610914893f23b1fa3f08af55b1875065dd2ecc7e6"} Oct 29 22:45:02.284235 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.284214 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerStarted","Data":"e2683d20653c6e508ffe6ffd21fa3b3fa8ce13f2b284107a79e851313bdb87e5"} Oct 29 22:45:02.324404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.324328 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" podStartSLOduration=2.324307609 podStartE2EDuration="2.324307609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.323094797 +0000 UTC m=+1817.638488877" watchObservedRunningTime="2026-10-29 22:45:02.324307609 +0000 UTC m=+1817.639701691" Oct 29 22:45:02.375503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.375456 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" podStartSLOduration=2.375442566 podStartE2EDuration="2.375442566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.370799822 +0000 UTC m=+1817.686193928" watchObservedRunningTime="2026-10-29 22:45:02.375442566 +0000 UTC m=+1817.690836647" Oct 29 22:45:02.465717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.462996 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" podStartSLOduration=2.462974525 podStartE2EDuration="2.462974525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.443583125 +0000 UTC m=+1817.758977206" watchObservedRunningTime="2026-10-29 22:45:02.462974525 +0000 UTC m=+1817.778368606" Oct 29 22:45:02.503696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.503620 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" podStartSLOduration=2.503598782 podStartE2EDuration="2.503598782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.486617945 +0000 UTC m=+1817.802012028" watchObservedRunningTime="2026-10-29 22:45:02.503598782 +0000 UTC m=+1817.818992864" Oct 29 22:45:02.533265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.533171 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" podStartSLOduration=2.533140813 podStartE2EDuration="2.533140813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.532508675 +0000 UTC m=+1817.847902756" watchObservedRunningTime="2026-10-29 22:45:02.533140813 +0000 UTC m=+1817.848534894" Oct 29 22:45:02.580408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.580352 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" podStartSLOduration=2.580333713 podStartE2EDuration="2.580333713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.577279883 +0000 UTC m=+1817.892673964" watchObservedRunningTime="2026-10-29 22:45:02.580333713 +0000 UTC m=+1817.895727796" Oct 29 22:45:02.676344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.676276 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" podStartSLOduration=2.676257696 podStartE2EDuration="2.676257696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.627132572 +0000 UTC m=+1817.942526678" watchObservedRunningTime="2026-10-29 22:45:02.676257696 +0000 UTC m=+1817.991651777" Oct 29 22:45:02.676893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:02.676856 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" podStartSLOduration=2.676846152 podStartE2EDuration="2.676846152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.676493861 +0000 UTC m=+1817.991887942" watchObservedRunningTime="2026-10-29 22:45:02.676846152 +0000 UTC m=+1817.992240255" Oct 29 22:45:03.293365 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.293325 5073 generic.go:356] "Generic (PLEG): container finished" podID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerID="f95e34909c0d3fff87053976c70c2163d84c62bf97d0ca959cf0805f4d110e5b" exitCode=60 Oct 29 22:45:03.293817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.293366 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerDied","Data":"f95e34909c0d3fff87053976c70c2163d84c62bf97d0ca959cf0805f4d110e5b"} Oct 29 22:45:03.293817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.293772 5073 scope.go:122] "RemoveContainer" containerID="f95e34909c0d3fff87053976c70c2163d84c62bf97d0ca959cf0805f4d110e5b" Oct 29 22:45:03.295291 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.295265 5073 generic.go:356] "Generic (PLEG): container finished" podID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerID="b0491176be563cda4575f63610914893f23b1fa3f08af55b1875065dd2ecc7e6" exitCode=60 Oct 29 22:45:03.295417 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.295351 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerDied","Data":"b0491176be563cda4575f63610914893f23b1fa3f08af55b1875065dd2ecc7e6"} Oct 29 22:45:03.295690 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.295593 5073 scope.go:122] "RemoveContainer" containerID="b0491176be563cda4575f63610914893f23b1fa3f08af55b1875065dd2ecc7e6" Oct 29 22:45:03.297128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.297105 5073 generic.go:356] "Generic (PLEG): container finished" podID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerID="e2683d20653c6e508ffe6ffd21fa3b3fa8ce13f2b284107a79e851313bdb87e5" exitCode=60 Oct 29 22:45:03.298374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.297227 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerDied","Data":"e2683d20653c6e508ffe6ffd21fa3b3fa8ce13f2b284107a79e851313bdb87e5"} Oct 29 22:45:03.298374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.297427 5073 scope.go:122] "RemoveContainer" containerID="e2683d20653c6e508ffe6ffd21fa3b3fa8ce13f2b284107a79e851313bdb87e5" Oct 29 22:45:03.299304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.299287 5073 generic.go:356] "Generic (PLEG): container finished" podID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerID="867a661cc857a6890e87a41f28916f9ca9f744dfe58dbf88f2e35f8de140e09c" exitCode=60 Oct 29 22:45:03.299488 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.299362 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerDied","Data":"867a661cc857a6890e87a41f28916f9ca9f744dfe58dbf88f2e35f8de140e09c"} Oct 29 22:45:03.299650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.299577 5073 scope.go:122] "RemoveContainer" containerID="867a661cc857a6890e87a41f28916f9ca9f744dfe58dbf88f2e35f8de140e09c" Oct 29 22:45:03.300922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.300902 5073 generic.go:356] "Generic (PLEG): container finished" podID="594df621-447e-4232-ab84-67bad2b0b940" containerID="6c31110ade0478e2805928de78990c5043f71dacf22da5ec7a1124aced785d41" exitCode=60 Oct 29 22:45:03.301008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.300966 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerDied","Data":"6c31110ade0478e2805928de78990c5043f71dacf22da5ec7a1124aced785d41"} Oct 29 22:45:03.301180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.301164 5073 scope.go:122] "RemoveContainer" containerID="6c31110ade0478e2805928de78990c5043f71dacf22da5ec7a1124aced785d41" Oct 29 22:45:03.302623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.302605 5073 generic.go:356] "Generic (PLEG): container finished" podID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerID="a640644309345b984e76580e2d9a0bb94e9842d4e19750e3daa49f54162f053c" exitCode=60 Oct 29 22:45:03.302707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.302659 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerDied","Data":"a640644309345b984e76580e2d9a0bb94e9842d4e19750e3daa49f54162f053c"} Oct 29 22:45:03.302902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.302884 5073 scope.go:122] "RemoveContainer" containerID="a640644309345b984e76580e2d9a0bb94e9842d4e19750e3daa49f54162f053c" Oct 29 22:45:03.304333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.304307 5073 generic.go:356] "Generic (PLEG): container finished" podID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerID="cc41784ae1941454aed42b2d016fa3464899bc91e1a220c9f1769d32ee16188e" exitCode=60 Oct 29 22:45:03.304426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.304378 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerDied","Data":"cc41784ae1941454aed42b2d016fa3464899bc91e1a220c9f1769d32ee16188e"} Oct 29 22:45:03.304582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.304564 5073 scope.go:122] "RemoveContainer" containerID="cc41784ae1941454aed42b2d016fa3464899bc91e1a220c9f1769d32ee16188e" Oct 29 22:45:03.312183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.307645 5073 generic.go:356] "Generic (PLEG): container finished" podID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerID="b75fec6a3d6abb4890be392fdc29513e961e52c268f37e21ee2ff011fc5824a8" exitCode=60 Oct 29 22:45:03.312183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.307682 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerDied","Data":"b75fec6a3d6abb4890be392fdc29513e961e52c268f37e21ee2ff011fc5824a8"} Oct 29 22:45:03.312183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:03.308034 5073 scope.go:122] "RemoveContainer" containerID="b75fec6a3d6abb4890be392fdc29513e961e52c268f37e21ee2ff011fc5824a8" Oct 29 22:45:04.327626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.327571 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerStarted","Data":"bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7"} Oct 29 22:45:04.329928 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.329881 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerStarted","Data":"a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789"} Oct 29 22:45:04.332264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.332194 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerStarted","Data":"4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82"} Oct 29 22:45:04.334322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.334305 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerStarted","Data":"4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f"} Oct 29 22:45:04.336407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.336381 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerStarted","Data":"8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a"} Oct 29 22:45:04.338414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.338388 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerStarted","Data":"b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337"} Oct 29 22:45:04.340203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.340179 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerStarted","Data":"0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2"} Oct 29 22:45:04.342670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:04.342647 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerStarted","Data":"5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b"} Oct 29 22:45:05.361140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.361104 5073 generic.go:356] "Generic (PLEG): container finished" podID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerID="b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337" exitCode=60 Oct 29 22:45:05.361652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.361195 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerDied","Data":"b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337"} Oct 29 22:45:05.361652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.361300 5073 scope.go:122] "RemoveContainer" containerID="cc41784ae1941454aed42b2d016fa3464899bc91e1a220c9f1769d32ee16188e" Oct 29 22:45:05.361652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.361464 5073 scope.go:122] "RemoveContainer" containerID="b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337" Oct 29 22:45:05.361833 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.361808 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn_odh-ai-gateway-infra(95ccf14b-4022-4e3a-9757-458e77b0c827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" Oct 29 22:45:05.363671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.363645 5073 generic.go:356] "Generic (PLEG): container finished" podID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerID="0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2" exitCode=60 Oct 29 22:45:05.363803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.363722 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerDied","Data":"0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2"} Oct 29 22:45:05.363930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.363914 5073 scope.go:122] "RemoveContainer" containerID="0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2" Oct 29 22:45:05.364272 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.364250 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-bxr4p_odh-ai-gateway-infra(37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" Oct 29 22:45:05.365461 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.365438 5073 generic.go:356] "Generic (PLEG): container finished" podID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerID="5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b" exitCode=60 Oct 29 22:45:05.365562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.365515 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerDied","Data":"5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b"} Oct 29 22:45:05.365730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.365702 5073 scope.go:122] "RemoveContainer" containerID="5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b" Oct 29 22:45:05.365954 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.365925 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl_odh-ai-gateway-infra(103d267a-055f-4afe-b5f7-63ea9dca3525)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" Oct 29 22:45:05.368623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.367966 5073 generic.go:356] "Generic (PLEG): container finished" podID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerID="bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7" exitCode=60 Oct 29 22:45:05.368623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.368074 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerDied","Data":"bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7"} Oct 29 22:45:05.368623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.368364 5073 scope.go:122] "RemoveContainer" containerID="bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7" Oct 29 22:45:05.368854 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.368622 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml_odh-ai-gateway-infra(ce1aed31-3ae5-409e-a08c-2f1804124a3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" Oct 29 22:45:05.370547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.370522 5073 generic.go:356] "Generic (PLEG): container finished" podID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerID="a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789" exitCode=60 Oct 29 22:45:05.370654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.370610 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerDied","Data":"a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789"} Oct 29 22:45:05.370799 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.370784 5073 scope.go:122] "RemoveContainer" containerID="a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789" Oct 29 22:45:05.371520 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.370989 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t_odh-ai-gateway-infra(54d6c86d-53c7-47ce-9597-d84fac5f3ae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" Oct 29 22:45:05.375181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.375144 5073 generic.go:356] "Generic (PLEG): container finished" podID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerID="4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82" exitCode=60 Oct 29 22:45:05.375296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.375237 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerDied","Data":"4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82"} Oct 29 22:45:05.375433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.375413 5073 scope.go:122] "RemoveContainer" containerID="4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82" Oct 29 22:45:05.375672 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.375649 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m_odh-ai-gateway-infra(8076aef0-abbb-4aba-9d5f-6afcabe62e28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" Oct 29 22:45:05.377222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.377199 5073 generic.go:356] "Generic (PLEG): container finished" podID="594df621-447e-4232-ab84-67bad2b0b940" containerID="4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f" exitCode=60 Oct 29 22:45:05.377339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.377277 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerDied","Data":"4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f"} Oct 29 22:45:05.377502 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.377484 5073 scope.go:122] "RemoveContainer" containerID="4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f" Oct 29 22:45:05.377740 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.377718 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh_odh-ai-gateway-infra(594df621-447e-4232-ab84-67bad2b0b940)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" podUID="594df621-447e-4232-ab84-67bad2b0b940" Oct 29 22:45:05.379565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.379542 5073 generic.go:356] "Generic (PLEG): container finished" podID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerID="8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a" exitCode=60 Oct 29 22:45:05.379653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.379576 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerDied","Data":"8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a"} Oct 29 22:45:05.379866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.379842 5073 scope.go:122] "RemoveContainer" containerID="8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a" Oct 29 22:45:05.380074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.380035 5073 scope.go:122] "RemoveContainer" containerID="b75fec6a3d6abb4890be392fdc29513e961e52c268f37e21ee2ff011fc5824a8" Oct 29 22:45:05.380128 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:05.380083 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd_odh-ai-gateway-infra(c7029b8b-85ef-4cde-8e00-f592b71056a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" Oct 29 22:45:05.400403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.400388 5073 scope.go:122] "RemoveContainer" containerID="f95e34909c0d3fff87053976c70c2163d84c62bf97d0ca959cf0805f4d110e5b" Oct 29 22:45:05.414033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.413996 5073 scope.go:122] "RemoveContainer" containerID="b0491176be563cda4575f63610914893f23b1fa3f08af55b1875065dd2ecc7e6" Oct 29 22:45:05.427585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.427568 5073 scope.go:122] "RemoveContainer" containerID="e2683d20653c6e508ffe6ffd21fa3b3fa8ce13f2b284107a79e851313bdb87e5" Oct 29 22:45:05.440292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.440259 5073 scope.go:122] "RemoveContainer" containerID="867a661cc857a6890e87a41f28916f9ca9f744dfe58dbf88f2e35f8de140e09c" Oct 29 22:45:05.457193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.456599 5073 scope.go:122] "RemoveContainer" containerID="6c31110ade0478e2805928de78990c5043f71dacf22da5ec7a1124aced785d41" Oct 29 22:45:05.471827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:05.471810 5073 scope.go:122] "RemoveContainer" containerID="a640644309345b984e76580e2d9a0bb94e9842d4e19750e3daa49f54162f053c" Oct 29 22:45:06.389484 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.389453 5073 scope.go:122] "RemoveContainer" containerID="a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789" Oct 29 22:45:06.389966 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.389806 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t_odh-ai-gateway-infra(54d6c86d-53c7-47ce-9597-d84fac5f3ae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" Oct 29 22:45:06.391295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.391267 5073 scope.go:122] "RemoveContainer" containerID="4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82" Oct 29 22:45:06.391563 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.391541 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m_odh-ai-gateway-infra(8076aef0-abbb-4aba-9d5f-6afcabe62e28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" Oct 29 22:45:06.392927 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.392907 5073 scope.go:122] "RemoveContainer" containerID="4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f" Oct 29 22:45:06.393245 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.393226 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh_odh-ai-gateway-infra(594df621-447e-4232-ab84-67bad2b0b940)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" podUID="594df621-447e-4232-ab84-67bad2b0b940" Oct 29 22:45:06.394536 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.394510 5073 scope.go:122] "RemoveContainer" containerID="8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a" Oct 29 22:45:06.394787 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.394769 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd_odh-ai-gateway-infra(c7029b8b-85ef-4cde-8e00-f592b71056a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" Oct 29 22:45:06.396091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.396072 5073 scope.go:122] "RemoveContainer" containerID="b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337" Oct 29 22:45:06.396352 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.396330 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn_odh-ai-gateway-infra(95ccf14b-4022-4e3a-9757-458e77b0c827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" Oct 29 22:45:06.398231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.398213 5073 scope.go:122] "RemoveContainer" containerID="0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2" Oct 29 22:45:06.398468 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.398449 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-bxr4p_odh-ai-gateway-infra(37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" Oct 29 22:45:06.399785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.399767 5073 scope.go:122] "RemoveContainer" containerID="5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b" Oct 29 22:45:06.400021 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.399999 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl_odh-ai-gateway-infra(103d267a-055f-4afe-b5f7-63ea9dca3525)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" Oct 29 22:45:06.401368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:06.401354 5073 scope.go:122] "RemoveContainer" containerID="bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7" Oct 29 22:45:06.401637 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:06.401619 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml_odh-ai-gateway-infra(ce1aed31-3ae5-409e-a08c-2f1804124a3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" Oct 29 22:45:09.255675 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:09.255633 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:45:11.905311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.905276 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk"] Oct 29 22:45:11.912588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.912560 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:11.930686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.930658 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-139eb7\"/\"openshift-service-ca.crt\"" Oct 29 22:45:11.931023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.931007 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-139eb7\"/\"kube-root-ca.crt\"" Oct 29 22:45:11.956978 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.956946 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk"] Oct 29 22:45:11.957708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.957687 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-139eb7\"/\"test-model-139eb7-kserve-self-signed-certs\"" Oct 29 22:45:11.957827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:11.957814 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-139eb7\"/\"default-dockercfg-8vkw9\"" Oct 29 22:45:12.055935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.055895 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.056085 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.056049 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.056131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.056085 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.056252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.056231 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.056286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.056265 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.056332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.056315 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpz8x\" (UniqueName: \"kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158070 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158372 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158233 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158372 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158274 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158373 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158407 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158453 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpz8x\" (UniqueName: \"kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158766 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.158895 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.158860 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.159078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.159037 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.160922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.160900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.161520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.161495 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.180814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.180748 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpz8x\" (UniqueName: \"kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x\") pod \"test-model-139eb7-kserve-68dbc897b5-cklxk\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.232176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.230666 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:12.924685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:12.924656 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk"] Oct 29 22:45:13.577568 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:13.577527 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" event={"ID":"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516","Type":"ContainerStarted","Data":"91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082"} Oct 29 22:45:13.577766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:13.577600 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" event={"ID":"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516","Type":"ContainerStarted","Data":"66a5d7debafa7570465c21444dab48740144d10f49abd4e178bac5aa95b2a61c"} Oct 29 22:45:13.631389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:13.631320 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" podStartSLOduration=2.631301166 podStartE2EDuration="2.631301166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:13.618513603 +0000 UTC m=+1828.933907685" watchObservedRunningTime="2026-10-29 22:45:13.631301166 +0000 UTC m=+1828.946695245" Oct 29 22:45:14.230856 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:14.230824 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:15.831065 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:15.831031 5073 scope.go:122] "RemoveContainer" containerID="b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337" Oct 29 22:45:15.831986 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:15.831716 5073 scope.go:122] "RemoveContainer" containerID="4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82" Oct 29 22:45:16.606130 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:16.606098 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerStarted","Data":"3753946dc061edcaa102bbf5b62d75a5fef5f86ed57ec53c806f3036ecdfb959"} Oct 29 22:45:16.609755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:16.609724 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerStarted","Data":"d9c85ceb8118bf3c7b7a240f6dca2104b1dabb5de128496cc49fd4be9addd425"} Oct 29 22:45:16.834030 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:16.834003 5073 scope.go:122] "RemoveContainer" containerID="a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789" Oct 29 22:45:16.835233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:16.835212 5073 scope.go:122] "RemoveContainer" containerID="bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7" Oct 29 22:45:17.620614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.620579 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerStarted","Data":"4950e5865ea6021f6ced2afb031a3a3bde8887a055f8bf60cb5be5b86d8bd4c6"} Oct 29 22:45:17.622757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.622726 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerStarted","Data":"483e931a6aad20d376986b947b08aed5cd4d703f5db8e590234081eedccc28b8"} Oct 29 22:45:17.624624 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.624597 5073 generic.go:356] "Generic (PLEG): container finished" podID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerID="d9c85ceb8118bf3c7b7a240f6dca2104b1dabb5de128496cc49fd4be9addd425" exitCode=60 Oct 29 22:45:17.624742 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.624669 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerDied","Data":"d9c85ceb8118bf3c7b7a240f6dca2104b1dabb5de128496cc49fd4be9addd425"} Oct 29 22:45:17.624805 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.624727 5073 scope.go:122] "RemoveContainer" containerID="4be181524c72897537ce135e343366aae19d55f8233d1497bc845721b4bb4c82" Oct 29 22:45:17.625085 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.625068 5073 scope.go:122] "RemoveContainer" containerID="d9c85ceb8118bf3c7b7a240f6dca2104b1dabb5de128496cc49fd4be9addd425" Oct 29 22:45:17.625424 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:17.625398 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m_odh-ai-gateway-infra(8076aef0-abbb-4aba-9d5f-6afcabe62e28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" Oct 29 22:45:17.626780 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.626761 5073 generic.go:356] "Generic (PLEG): container finished" podID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerID="3753946dc061edcaa102bbf5b62d75a5fef5f86ed57ec53c806f3036ecdfb959" exitCode=60 Oct 29 22:45:17.626868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.626842 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerDied","Data":"3753946dc061edcaa102bbf5b62d75a5fef5f86ed57ec53c806f3036ecdfb959"} Oct 29 22:45:17.627142 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.627128 5073 scope.go:122] "RemoveContainer" containerID="3753946dc061edcaa102bbf5b62d75a5fef5f86ed57ec53c806f3036ecdfb959" Oct 29 22:45:17.627377 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:17.627359 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn_odh-ai-gateway-infra(95ccf14b-4022-4e3a-9757-458e77b0c827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" Oct 29 22:45:17.638761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.638734 5073 scope.go:122] "RemoveContainer" containerID="b4b99604788962f86582207d9d3f9c980f5abc5f33edc227ecfff718e82d0337" Oct 29 22:45:17.659333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.659312 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn"] Oct 29 22:45:17.701050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.701027 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m"] Oct 29 22:45:17.831810 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.831781 5073 scope.go:122] "RemoveContainer" containerID="5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b" Oct 29 22:45:17.832015 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.831996 5073 scope.go:122] "RemoveContainer" containerID="0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2" Oct 29 22:45:17.832354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.832332 5073 scope.go:122] "RemoveContainer" containerID="4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f" Oct 29 22:45:17.832477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:17.832462 5073 scope.go:122] "RemoveContainer" containerID="8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a" Oct 29 22:45:18.636489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.636458 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerStarted","Data":"ebc354b48ec317ea32243e143a863afbabe8aea1b183a54c3914613e6cb4219a"} Oct 29 22:45:18.640892 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.640866 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerStarted","Data":"3b3b6804f534939f8cace0d94d8418002d8268364d3fde7fc4bb4e9714a00724"} Oct 29 22:45:18.643295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.643272 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerStarted","Data":"f9d3018d1f838328823dfb41dd4b0eb0ee33ae6f5b10cc1558a833c4ae6a0af5"} Oct 29 22:45:18.645467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.645444 5073 generic.go:356] "Generic (PLEG): container finished" podID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerID="4950e5865ea6021f6ced2afb031a3a3bde8887a055f8bf60cb5be5b86d8bd4c6" exitCode=60 Oct 29 22:45:18.645583 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.645527 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerDied","Data":"4950e5865ea6021f6ced2afb031a3a3bde8887a055f8bf60cb5be5b86d8bd4c6"} Oct 29 22:45:18.645647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.645594 5073 scope.go:122] "RemoveContainer" containerID="bb66233aefe1f098e00cd95f261fcf28ab9a40ceddc20fbfb3f2699665a737c7" Oct 29 22:45:18.646002 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.645984 5073 scope.go:122] "RemoveContainer" containerID="4950e5865ea6021f6ced2afb031a3a3bde8887a055f8bf60cb5be5b86d8bd4c6" Oct 29 22:45:18.646419 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:18.646395 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml_odh-ai-gateway-infra(ce1aed31-3ae5-409e-a08c-2f1804124a3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" Oct 29 22:45:18.647944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.647922 5073 generic.go:356] "Generic (PLEG): container finished" podID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerID="483e931a6aad20d376986b947b08aed5cd4d703f5db8e590234081eedccc28b8" exitCode=60 Oct 29 22:45:18.648050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.647980 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerDied","Data":"483e931a6aad20d376986b947b08aed5cd4d703f5db8e590234081eedccc28b8"} Oct 29 22:45:18.648354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.648327 5073 scope.go:122] "RemoveContainer" containerID="483e931a6aad20d376986b947b08aed5cd4d703f5db8e590234081eedccc28b8" Oct 29 22:45:18.648692 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:18.648670 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t_odh-ai-gateway-infra(54d6c86d-53c7-47ce-9597-d84fac5f3ae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" Oct 29 22:45:18.654014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.653989 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerStarted","Data":"46a023da6177e28d3ff500514a222a2ecaf2ef374d2739677b89ca04ed9f06e7"} Oct 29 22:45:18.662430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.662410 5073 scope.go:122] "RemoveContainer" containerID="a1e907d49963b708289f27a50e414a1921fda9f26954ca6b614ba9a61f9d1789" Oct 29 22:45:18.686258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.685259 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml"] Oct 29 22:45:18.725493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.723614 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t"] Oct 29 22:45:18.914422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.914389 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:18.974621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.974591 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:18.997179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.997118 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca\") pod \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " Oct 29 22:45:18.997359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.997339 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca\") pod \"95ccf14b-4022-4e3a-9757-458e77b0c827\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " Oct 29 22:45:18.997474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.997457 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq\" (UniqueName: \"kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq\") pod \"95ccf14b-4022-4e3a-9757-458e77b0c827\" (UID: \"95ccf14b-4022-4e3a-9757-458e77b0c827\") " Oct 29 22:45:18.997843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.997822 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca" pod "95ccf14b-4022-4e3a-9757-458e77b0c827" (UID: "95ccf14b-4022-4e3a-9757-458e77b0c827"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.997987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.997929 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca" pod "8076aef0-abbb-4aba-9d5f-6afcabe62e28" (UID: "8076aef0-abbb-4aba-9d5f-6afcabe62e28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.998127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.998071 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm\" (UniqueName: \"kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm\") pod \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\" (UID: \"8076aef0-abbb-4aba-9d5f-6afcabe62e28\") " Oct 29 22:45:18.998907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.998882 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8076aef0-abbb-4aba-9d5f-6afcabe62e28-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.999008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:18.998913 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ccf14b-4022-4e3a-9757-458e77b0c827-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.000826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.000801 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm" pod "8076aef0-abbb-4aba-9d5f-6afcabe62e28" (UID: "8076aef0-abbb-4aba-9d5f-6afcabe62e28"). InnerVolumeSpecName "kube-api-access-bfkvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.000952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.000830 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq" pod "95ccf14b-4022-4e3a-9757-458e77b0c827" (UID: "95ccf14b-4022-4e3a-9757-458e77b0c827"). InnerVolumeSpecName "kube-api-access-spgmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.100354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.100275 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spgmq\" (UniqueName: \"kubernetes.io/projected/95ccf14b-4022-4e3a-9757-458e77b0c827-kube-api-access-spgmq\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.100354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.100312 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkvm\" (UniqueName: \"kubernetes.io/projected/8076aef0-abbb-4aba-9d5f-6afcabe62e28-kube-api-access-bfkvm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.665319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.665291 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" Oct 29 22:45:19.665319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.665304 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m" event={"ID":"8076aef0-abbb-4aba-9d5f-6afcabe62e28","Type":"ContainerDied","Data":"c5792c25e3ced37a52b6506e5be0f50b8cb3875e1b1857b09cd2101db9afed50"} Oct 29 22:45:19.665974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.665371 5073 scope.go:122] "RemoveContainer" containerID="d9c85ceb8118bf3c7b7a240f6dca2104b1dabb5de128496cc49fd4be9addd425" Oct 29 22:45:19.669420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.668085 5073 generic.go:356] "Generic (PLEG): container finished" podID="594df621-447e-4232-ab84-67bad2b0b940" containerID="46a023da6177e28d3ff500514a222a2ecaf2ef374d2739677b89ca04ed9f06e7" exitCode=60 Oct 29 22:45:19.669420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.668219 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerDied","Data":"46a023da6177e28d3ff500514a222a2ecaf2ef374d2739677b89ca04ed9f06e7"} Oct 29 22:45:19.669420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.668653 5073 scope.go:122] "RemoveContainer" containerID="46a023da6177e28d3ff500514a222a2ecaf2ef374d2739677b89ca04ed9f06e7" Oct 29 22:45:19.669420 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:19.669041 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh_odh-ai-gateway-infra(594df621-447e-4232-ab84-67bad2b0b940)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" podUID="594df621-447e-4232-ab84-67bad2b0b940" Oct 29 22:45:19.671660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.671638 5073 generic.go:356] "Generic (PLEG): container finished" podID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerID="ebc354b48ec317ea32243e143a863afbabe8aea1b183a54c3914613e6cb4219a" exitCode=60 Oct 29 22:45:19.671777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.671711 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerDied","Data":"ebc354b48ec317ea32243e143a863afbabe8aea1b183a54c3914613e6cb4219a"} Oct 29 22:45:19.672134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.672116 5073 scope.go:122] "RemoveContainer" containerID="ebc354b48ec317ea32243e143a863afbabe8aea1b183a54c3914613e6cb4219a" Oct 29 22:45:19.672483 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:19.672461 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd_odh-ai-gateway-infra(c7029b8b-85ef-4cde-8e00-f592b71056a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" Oct 29 22:45:19.673269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.673246 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" Oct 29 22:45:19.673387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.673243 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn" event={"ID":"95ccf14b-4022-4e3a-9757-458e77b0c827","Type":"ContainerDied","Data":"9104719cc62c29c5696861d7a4ce0352d5f1670197c5572e40b45bdf8c73b434"} Oct 29 22:45:19.676662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.676634 5073 generic.go:356] "Generic (PLEG): container finished" podID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerID="3b3b6804f534939f8cace0d94d8418002d8268364d3fde7fc4bb4e9714a00724" exitCode=60 Oct 29 22:45:19.676800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.676777 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerDied","Data":"3b3b6804f534939f8cace0d94d8418002d8268364d3fde7fc4bb4e9714a00724"} Oct 29 22:45:19.677511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.677178 5073 scope.go:122] "RemoveContainer" containerID="3b3b6804f534939f8cace0d94d8418002d8268364d3fde7fc4bb4e9714a00724" Oct 29 22:45:19.677602 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:19.677576 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-bxr4p_odh-ai-gateway-infra(37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" Oct 29 22:45:19.679144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.679118 5073 generic.go:356] "Generic (PLEG): container finished" podID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerID="f9d3018d1f838328823dfb41dd4b0eb0ee33ae6f5b10cc1558a833c4ae6a0af5" exitCode=60 Oct 29 22:45:19.679345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.679324 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerDied","Data":"f9d3018d1f838328823dfb41dd4b0eb0ee33ae6f5b10cc1558a833c4ae6a0af5"} Oct 29 22:45:19.679657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.679638 5073 scope.go:122] "RemoveContainer" containerID="f9d3018d1f838328823dfb41dd4b0eb0ee33ae6f5b10cc1558a833c4ae6a0af5" Oct 29 22:45:19.679923 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:19.679901 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl_odh-ai-gateway-infra(103d267a-055f-4afe-b5f7-63ea9dca3525)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" Oct 29 22:45:19.680556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.680541 5073 scope.go:122] "RemoveContainer" containerID="4b92cb0781093271f5a54f774ae1dfecb3d890367074ae192324df6a8b8c554f" Oct 29 22:45:19.696681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.696663 5073 scope.go:122] "RemoveContainer" containerID="8a893d05b03a3df001d3d6fb691f5aa1e6f0bb88522cdaec289853f1d503835a" Oct 29 22:45:19.724825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.719893 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd"] Oct 29 22:45:19.729118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.729084 5073 scope.go:122] "RemoveContainer" containerID="3753946dc061edcaa102bbf5b62d75a5fef5f86ed57ec53c806f3036ecdfb959" Oct 29 22:45:19.749298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.749124 5073 scope.go:122] "RemoveContainer" containerID="0a41323520f8671f406035eab71ff9275c66c3740e9a98d1e48094cdd1a132b2" Oct 29 22:45:19.773194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.771658 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl"] Oct 29 22:45:19.788550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.788197 5073 scope.go:122] "RemoveContainer" containerID="5baf31b2401bdc6d82a8ba49a8fb442c1b40d132afb89f98cd62bb620a498e2b" Oct 29 22:45:19.874048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.873978 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm"] Oct 29 22:45:19.874279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.874253 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" containerID="cri-o://36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562" gracePeriod=30 Oct 29 22:45:19.918703 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.918619 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l"] Oct 29 22:45:19.918888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.918855 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" containerID="cri-o://98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d" gracePeriod=30 Oct 29 22:45:19.933224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.933198 5073 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:19.933427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.933271 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:19.935735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.935711 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj"] Oct 29 22:45:19.935967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:19.935932 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" containerID="cri-o://51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467" gracePeriod=30 Oct 29 22:45:20.097826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.097733 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:20.111700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.108039 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p"] Oct 29 22:45:20.212224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.211901 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:20.249173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.248371 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m"] Oct 29 22:45:20.253900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.253847 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2\" (UniqueName: \"kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2\") pod \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " Oct 29 22:45:20.267559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.267521 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca\") pod \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " Oct 29 22:45:20.267812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.267791 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca\") pod \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\" (UID: \"54d6c86d-53c7-47ce-9597-d84fac5f3ae8\") " Oct 29 22:45:20.268661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.268636 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca" pod "ce1aed31-3ae5-409e-a08c-2f1804124a3b" (UID: "ce1aed31-3ae5-409e-a08c-2f1804124a3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.269021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.268987 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca" pod "54d6c86d-53c7-47ce-9597-d84fac5f3ae8" (UID: "54d6c86d-53c7-47ce-9597-d84fac5f3ae8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.269021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.269013 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1aed31-3ae5-409e-a08c-2f1804124a3b-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.303445 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.298434 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2" pod "54d6c86d-53c7-47ce-9597-d84fac5f3ae8" (UID: "54d6c86d-53c7-47ce-9597-d84fac5f3ae8"). InnerVolumeSpecName "kube-api-access-bxvh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.309387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.309317 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e81b56-29888565-gpx2m"] Oct 29 22:45:20.352282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.352247 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh"] Oct 29 22:45:20.369930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.369860 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7\" (UniqueName: \"kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7\") pod \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\" (UID: \"ce1aed31-3ae5-409e-a08c-2f1804124a3b\") " Oct 29 22:45:20.370992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.370960 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.371121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.370995 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvh2\" (UniqueName: \"kubernetes.io/projected/54d6c86d-53c7-47ce-9597-d84fac5f3ae8-kube-api-access-bxvh2\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.376811 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.376750 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn"] Oct 29 22:45:20.377144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.377113 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7" pod "ce1aed31-3ae5-409e-a08c-2f1804124a3b" (UID: "ce1aed31-3ae5-409e-a08c-2f1804124a3b"). InnerVolumeSpecName "kube-api-access-jfvs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.387509 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.387480 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a558b3-29888565-dx2tn"] Oct 29 22:45:20.480360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.478570 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfvs7\" (UniqueName: \"kubernetes.io/projected/ce1aed31-3ae5-409e-a08c-2f1804124a3b-kube-api-access-jfvs7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.720236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.720208 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" Oct 29 22:45:20.720789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.720652 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml" event={"ID":"ce1aed31-3ae5-409e-a08c-2f1804124a3b","Type":"ContainerDied","Data":"5920529a9a5821cc6a0af23618aa3bd5413bb80fd699275781db5c3aa963b557"} Oct 29 22:45:20.720919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.720902 5073 scope.go:122] "RemoveContainer" containerID="4950e5865ea6021f6ced2afb031a3a3bde8887a055f8bf60cb5be5b86d8bd4c6" Oct 29 22:45:20.725204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.724863 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" event={"ID":"54d6c86d-53c7-47ce-9597-d84fac5f3ae8","Type":"ContainerDied","Data":"02b7c0a076b597fc2d9fe676a200385dc5801437c5faa3c48c7120386c83b47e"} Oct 29 22:45:20.725204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.724975 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t" Oct 29 22:45:20.738906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.738885 5073 scope.go:122] "RemoveContainer" containerID="483e931a6aad20d376986b947b08aed5cd4d703f5db8e590234081eedccc28b8" Oct 29 22:45:20.857442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.855473 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" path="/var/lib/kubelet/pods/8076aef0-abbb-4aba-9d5f-6afcabe62e28/volumes" Oct 29 22:45:20.857442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.856034 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" path="/var/lib/kubelet/pods/95ccf14b-4022-4e3a-9757-458e77b0c827/volumes" Oct 29 22:45:20.925047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.925018 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t"] Oct 29 22:45:20.932271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.932246 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:20.946377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.946351 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36b46d-29888565-9hp8t"] Oct 29 22:45:20.998337 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:20.998249 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml"] Oct 29 22:45:21.001961 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.001925 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca\") pod \"c7029b8b-85ef-4cde-8e00-f592b71056a0\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " Oct 29 22:45:21.002664 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.002645 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb\" (UniqueName: \"kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb\") pod \"c7029b8b-85ef-4cde-8e00-f592b71056a0\" (UID: \"c7029b8b-85ef-4cde-8e00-f592b71056a0\") " Oct 29 22:45:21.003849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.002500 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca" pod "c7029b8b-85ef-4cde-8e00-f592b71056a0" (UID: "c7029b8b-85ef-4cde-8e00-f592b71056a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.005129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.004593 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0d12c5-29888565-h86ml"] Oct 29 22:45:21.017974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.017949 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb" pod "c7029b8b-85ef-4cde-8e00-f592b71056a0" (UID: "c7029b8b-85ef-4cde-8e00-f592b71056a0"). InnerVolumeSpecName "kube-api-access-rbjcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.105513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.105485 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7029b8b-85ef-4cde-8e00-f592b71056a0-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.105513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.105514 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjcb\" (UniqueName: \"kubernetes.io/projected/c7029b8b-85ef-4cde-8e00-f592b71056a0-kube-api-access-rbjcb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.112308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.112286 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:21.119106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.119086 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:21.123510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.123491 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:21.217875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.217814 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca\") pod \"103d267a-055f-4afe-b5f7-63ea9dca3525\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " Oct 29 22:45:21.218288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.218265 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca\") pod \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " Oct 29 22:45:21.218467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.218450 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8\" (UniqueName: \"kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8\") pod \"103d267a-055f-4afe-b5f7-63ea9dca3525\" (UID: \"103d267a-055f-4afe-b5f7-63ea9dca3525\") " Oct 29 22:45:21.218610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.218596 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv\" (UniqueName: \"kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv\") pod \"594df621-447e-4232-ab84-67bad2b0b940\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " Oct 29 22:45:21.218727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.218712 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca\") pod \"594df621-447e-4232-ab84-67bad2b0b940\" (UID: \"594df621-447e-4232-ab84-67bad2b0b940\") " Oct 29 22:45:21.219034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.219015 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w\" (UniqueName: \"kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w\") pod \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\" (UID: \"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6\") " Oct 29 22:45:21.221730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.221689 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca" pod "37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" (UID: "37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.225995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.223939 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w" pod "37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" (UID: "37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6"). InnerVolumeSpecName "kube-api-access-7p85w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.225995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.224910 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca" pod "103d267a-055f-4afe-b5f7-63ea9dca3525" (UID: "103d267a-055f-4afe-b5f7-63ea9dca3525"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.225995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.225177 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca" pod "594df621-447e-4232-ab84-67bad2b0b940" (UID: "594df621-447e-4232-ab84-67bad2b0b940"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.235674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.235369 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8" pod "103d267a-055f-4afe-b5f7-63ea9dca3525" (UID: "103d267a-055f-4afe-b5f7-63ea9dca3525"). InnerVolumeSpecName "kube-api-access-8hrr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.238088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.238064 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv" pod "594df621-447e-4232-ab84-67bad2b0b940" (UID: "594df621-447e-4232-ab84-67bad2b0b940"). InnerVolumeSpecName "kube-api-access-z2scv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.321602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.321521 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103d267a-055f-4afe-b5f7-63ea9dca3525-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.321602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.321553 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.321602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.321564 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hrr8\" (UniqueName: \"kubernetes.io/projected/103d267a-055f-4afe-b5f7-63ea9dca3525-kube-api-access-8hrr8\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.321602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.321574 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2scv\" (UniqueName: \"kubernetes.io/projected/594df621-447e-4232-ab84-67bad2b0b940-kube-api-access-z2scv\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.321602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.321583 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594df621-447e-4232-ab84-67bad2b0b940-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.321602 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.321591 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p85w\" (UniqueName: \"kubernetes.io/projected/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6-kube-api-access-7p85w\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.766782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.766751 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" Oct 29 22:45:21.767318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.766755 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh" event={"ID":"594df621-447e-4232-ab84-67bad2b0b940","Type":"ContainerDied","Data":"f17818bdb2efa7e900a4b416be5d700645bb7feee87114d886046a30b3591edf"} Oct 29 22:45:21.767318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.766892 5073 scope.go:122] "RemoveContainer" containerID="46a023da6177e28d3ff500514a222a2ecaf2ef374d2739677b89ca04ed9f06e7" Oct 29 22:45:21.768263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.768240 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" Oct 29 22:45:21.768366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.768240 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd" event={"ID":"c7029b8b-85ef-4cde-8e00-f592b71056a0","Type":"ContainerDied","Data":"8cceaf9403a7cd0dc717808d22de93033b0f056f4e32e456409d556804f5ec40"} Oct 29 22:45:21.769671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.769654 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" Oct 29 22:45:21.769787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.769694 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p" event={"ID":"37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6","Type":"ContainerDied","Data":"9043377a0da34b56168f008a6ed5888d47e416043c249d69eb4a74e31088ce1a"} Oct 29 22:45:21.772422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.772398 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" Oct 29 22:45:21.772613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.772526 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl" event={"ID":"103d267a-055f-4afe-b5f7-63ea9dca3525","Type":"ContainerDied","Data":"8bc7961c299f45f642a2e92b46f594e243fb5f6ba62ebab7a5c597d5d3d79a41"} Oct 29 22:45:21.781732 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.781711 5073 scope.go:122] "RemoveContainer" containerID="ebc354b48ec317ea32243e143a863afbabe8aea1b183a54c3914613e6cb4219a" Oct 29 22:45:21.792408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.792358 5073 scope.go:122] "RemoveContainer" containerID="3b3b6804f534939f8cace0d94d8418002d8268364d3fde7fc4bb4e9714a00724" Oct 29 22:45:21.802358 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.802334 5073 scope.go:122] "RemoveContainer" containerID="f9d3018d1f838328823dfb41dd4b0eb0ee33ae6f5b10cc1558a833c4ae6a0af5" Oct 29 22:45:21.862710 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.862682 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p"] Oct 29 22:45:21.876141 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.876119 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-bxr4p"] Oct 29 22:45:21.918287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.918264 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh"] Oct 29 22:45:21.927029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.927009 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-63db1a-29888565-vgdqh"] Oct 29 22:45:21.968470 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.968446 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl"] Oct 29 22:45:21.973555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:21.973530 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-40cf9a-29888565-w42bl"] Oct 29 22:45:22.007675 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.007652 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd"] Oct 29 22:45:22.011055 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.011024 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a26426-29888565-n96qd"] Oct 29 22:45:22.231679 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.231644 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:22.245081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.245058 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:22.803409 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.803378 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:45:22.840837 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.840803 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" path="/var/lib/kubelet/pods/103d267a-055f-4afe-b5f7-63ea9dca3525/volumes" Oct 29 22:45:22.841329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.841310 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" path="/var/lib/kubelet/pods/37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6/volumes" Oct 29 22:45:22.841757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.841744 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" path="/var/lib/kubelet/pods/54d6c86d-53c7-47ce-9597-d84fac5f3ae8/volumes" Oct 29 22:45:22.842534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.842519 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594df621-447e-4232-ab84-67bad2b0b940" path="/var/lib/kubelet/pods/594df621-447e-4232-ab84-67bad2b0b940/volumes" Oct 29 22:45:22.842961 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.842948 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" path="/var/lib/kubelet/pods/c7029b8b-85ef-4cde-8e00-f592b71056a0/volumes" Oct 29 22:45:22.843398 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:22.843383 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" path="/var/lib/kubelet/pods/ce1aed31-3ae5-409e-a08c-2f1804124a3b/volumes" Oct 29 22:45:25.260195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.260143 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:45:25.328144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.328112 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.328463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.328438 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.328677 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.328657 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.328816 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.328799 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.328946 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.328931 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.329232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.329212 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.329376 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.329359 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.329547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.329530 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.329697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.329681 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token\") pod \"303790fa-f5a1-46e8-a020-6520f938da55\" (UID: \"303790fa-f5a1-46e8-a020-6520f938da55\") " Oct 29 22:45:25.330902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.330794 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.331224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.331170 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.331345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.331323 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.332048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.331493 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.332048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.331755 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.332048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.331973 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.332048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.331993 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303790fa-f5a1-46e8-a020-6520f938da55-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.332048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.332017 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.332048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.332032 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.337632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.337589 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.340585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.340562 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.340831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.340804 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.341264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.341240 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9" pod "303790fa-f5a1-46e8-a020-6520f938da55" (UID: "303790fa-f5a1-46e8-a020-6520f938da55"). InnerVolumeSpecName "kube-api-access-5jtf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.380122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.380098 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:45:25.383673 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.383652 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:45:25.432764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.432739 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.432915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.432778 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.432915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.432804 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.432915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.432849 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.432915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.432887 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.433112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.432956 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.433112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433006 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.433112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433032 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.433112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433034 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.433112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433104 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.433460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433200 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.433460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433253 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.433460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433300 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.433460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433362 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.433460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433435 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.433884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433512 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.433884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433604 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.433884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433850 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.433884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.433871 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.434108 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434067 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.434108 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434080 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.434275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434204 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.434342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434277 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket\") pod \"711f9cff-d484-4ad7-a198-848987f51d17\" (UID: \"711f9cff-d484-4ad7-a198-848987f51d17\") " Oct 29 22:45:25.434342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434322 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data\") pod \"21adecb0-3bec-4eda-abcb-88f0d049577c\" (UID: \"21adecb0-3bec-4eda-abcb-88f0d049577c\") " Oct 29 22:45:25.434446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434207 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.434446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.434388 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.435293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435268 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435302 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtf9\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-kube-api-access-5jtf9\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435318 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21adecb0-3bec-4eda-abcb-88f0d049577c-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435332 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303790fa-f5a1-46e8-a020-6520f938da55-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435347 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435360 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435371 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435382 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303790fa-f5a1-46e8-a020-6520f938da55-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435393 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711f9cff-d484-4ad7-a198-848987f51d17-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435404 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435408 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435415 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303790fa-f5a1-46e8-a020-6520f938da55-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435426 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435437 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.435891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435664 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.435891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.435842 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.436214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.436139 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.436355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.436314 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.436645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.436579 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.437116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.437091 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "kube-api-access-jjfnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.437258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.437119 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token" pod "21adecb0-3bec-4eda-abcb-88f0d049577c" (UID: "21adecb0-3bec-4eda-abcb-88f0d049577c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.437485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.437458 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.438468 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.438447 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.438757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.438738 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv" pod "711f9cff-d484-4ad7-a198-848987f51d17" (UID: "711f9cff-d484-4ad7-a198-848987f51d17"). InnerVolumeSpecName "kube-api-access-qwfvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.537653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537621 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711f9cff-d484-4ad7-a198-848987f51d17-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537654 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537664 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537676 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537688 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711f9cff-d484-4ad7-a198-848987f51d17-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537696 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537704 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537711 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21adecb0-3bec-4eda-abcb-88f0d049577c-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537719 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwfvv\" (UniqueName: \"kubernetes.io/projected/711f9cff-d484-4ad7-a198-848987f51d17-kube-api-access-qwfvv\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.537901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.537728 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfnb\" (UniqueName: \"kubernetes.io/projected/21adecb0-3bec-4eda-abcb-88f0d049577c-kube-api-access-jjfnb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.815647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.815562 5073 generic.go:356] "Generic (PLEG): container finished" podID="711f9cff-d484-4ad7-a198-848987f51d17" containerID="36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562" exitCode=0 Oct 29 22:45:25.815647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.815611 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" Oct 29 22:45:25.815879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.815651 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" event={"ID":"711f9cff-d484-4ad7-a198-848987f51d17","Type":"ContainerDied","Data":"36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562"} Oct 29 22:45:25.815879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.815716 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm" event={"ID":"711f9cff-d484-4ad7-a198-848987f51d17","Type":"ContainerDied","Data":"4d47d55e4d75e873376b1c68e512e99970628e48f5c0185ce5558ab5c9b6fba5"} Oct 29 22:45:25.815879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.815749 5073 scope.go:122] "RemoveContainer" containerID="36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562" Oct 29 22:45:25.817515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.817487 5073 generic.go:356] "Generic (PLEG): container finished" podID="303790fa-f5a1-46e8-a020-6520f938da55" containerID="98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d" exitCode=0 Oct 29 22:45:25.817648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.817571 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" Oct 29 22:45:25.817648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.817601 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" event={"ID":"303790fa-f5a1-46e8-a020-6520f938da55","Type":"ContainerDied","Data":"98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d"} Oct 29 22:45:25.817648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.817640 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l" event={"ID":"303790fa-f5a1-46e8-a020-6520f938da55","Type":"ContainerDied","Data":"5625ec8f0825f84208aefbc58484c0de73ff2d19c2982c5a225cbe21f6e88a74"} Oct 29 22:45:25.819309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.819290 5073 generic.go:356] "Generic (PLEG): container finished" podID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerID="51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467" exitCode=0 Oct 29 22:45:25.819421 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.819346 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" Oct 29 22:45:25.819531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.819506 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" event={"ID":"21adecb0-3bec-4eda-abcb-88f0d049577c","Type":"ContainerDied","Data":"51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467"} Oct 29 22:45:25.819607 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.819547 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj" event={"ID":"21adecb0-3bec-4eda-abcb-88f0d049577c","Type":"ContainerDied","Data":"ba41447b588a57b6b480d060d3296a819f3842ebfa1b5a512f70a3e1d98dbb40"} Oct 29 22:45:25.830633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.830609 5073 scope.go:122] "RemoveContainer" containerID="36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562" Oct 29 22:45:25.831378 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:25.831345 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562\": container with ID starting with 36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562 not found: ID does not exist" containerID="36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562" Oct 29 22:45:25.831604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.831389 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562"} err="failed to get container status \"36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562\": rpc error: code = NotFound desc = could not find container \"36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562\": container with ID starting with 36570b7f30e4abc2072962c2a08ecb75fd9763a99f14636ce9a0a64395b43562 not found: ID does not exist" Oct 29 22:45:25.831604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.831580 5073 scope.go:122] "RemoveContainer" containerID="98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d" Oct 29 22:45:25.852987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.852969 5073 scope.go:122] "RemoveContainer" containerID="98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d" Oct 29 22:45:25.853336 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:25.853317 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d\": container with ID starting with 98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d not found: ID does not exist" containerID="98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d" Oct 29 22:45:25.853390 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.853344 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d"} err="failed to get container status \"98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d\": rpc error: code = NotFound desc = could not find container \"98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d\": container with ID starting with 98076dbcc399bd76eeefcba92ffa53a20e80b4156474f477938d121302488d6d not found: ID does not exist" Oct 29 22:45:25.853390 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.853361 5073 scope.go:122] "RemoveContainer" containerID="51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467" Oct 29 22:45:25.865842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.865820 5073 scope.go:122] "RemoveContainer" containerID="51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467" Oct 29 22:45:25.866196 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:25.866139 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467\": container with ID starting with 51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467 not found: ID does not exist" containerID="51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467" Oct 29 22:45:25.866296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.866206 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467"} err="failed to get container status \"51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467\": rpc error: code = NotFound desc = could not find container \"51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467\": container with ID starting with 51e38a197d8c520a309fa85fc654383c8d95fd2e672b6728ab63b0f085bbf467 not found: ID does not exist" Oct 29 22:45:25.877850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.877826 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm"] Oct 29 22:45:25.891006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.890984 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-hwhbm"] Oct 29 22:45:25.948889 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.948868 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj"] Oct 29 22:45:25.952391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.952369 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-72dnj"] Oct 29 22:45:25.989585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.989561 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l"] Oct 29 22:45:25.992817 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:25.992796 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-ffs2l"] Oct 29 22:45:26.843124 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:26.843096 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" path="/var/lib/kubelet/pods/21adecb0-3bec-4eda-abcb-88f0d049577c/volumes" Oct 29 22:45:26.843630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:26.843613 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303790fa-f5a1-46e8-a020-6520f938da55" path="/var/lib/kubelet/pods/303790fa-f5a1-46e8-a020-6520f938da55/volumes" Oct 29 22:45:26.844072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:26.844058 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711f9cff-d484-4ad7-a198-848987f51d17" path="/var/lib/kubelet/pods/711f9cff-d484-4ad7-a198-848987f51d17/volumes" Oct 29 22:45:39.344805 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:45:39.344764 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:45:48.811670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:45:48.811631 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:46:05.686656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.686610 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft"] Oct 29 22:46:05.687748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687726 5073 cpu_manager.go:414] "containerMap: removing container" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:05.687748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687750 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687759 5073 cpu_manager.go:414] "containerMap: removing container" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687764 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687776 5073 cpu_manager.go:414] "containerMap: removing container" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687780 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687789 5073 cpu_manager.go:414] "containerMap: removing container" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687793 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687803 5073 cpu_manager.go:414] "containerMap: removing container" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687807 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687812 5073 cpu_manager.go:414] "containerMap: removing container" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687816 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687821 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687825 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687833 5073 cpu_manager.go:414] "containerMap: removing container" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687838 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687845 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687850 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687860 5073 cpu_manager.go:414] "containerMap: removing container" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687865 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687886 5073 cpu_manager.go:414] "containerMap: removing container" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687890 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687897 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687901 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687913 5073 cpu_manager.go:414] "containerMap: removing container" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687917 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687924 5073 cpu_manager.go:414] "containerMap: removing container" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687928 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687934 5073 cpu_manager.go:414] "containerMap: removing container" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687940 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687946 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:05.687937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687951 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687959 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687964 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687972 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687976 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687988 5073 cpu_manager.go:414] "containerMap: removing container" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687992 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.687999 5073 cpu_manager.go:414] "containerMap: removing container" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688003 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688016 5073 cpu_manager.go:414] "containerMap: removing container" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688020 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688107 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688116 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="303790fa-f5a1-46e8-a020-6520f938da55" containerName="istio-proxy" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688123 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688130 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688136 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688141 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688164 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688173 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688180 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688187 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688195 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688202 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688213 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688221 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688231 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688239 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="711f9cff-d484-4ad7-a198-848987f51d17" containerName="istio-proxy" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688249 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688258 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688268 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="21adecb0-3bec-4eda-abcb-88f0d049577c" containerName="istio-proxy" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688277 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688285 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688294 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:05.689118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.688300 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="103d267a-055f-4afe-b5f7-63ea9dca3525" containerName="cleanup" Oct 29 22:46:05.692257 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.692240 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.731395 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.731303 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft"] Oct 29 22:46:05.739391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.739362 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6bc7b2-openshift-default-dockercfg-g8djs\"" Oct 29 22:46:05.776558 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.776525 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.776743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.776666 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.776743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.776729 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.776884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.776801 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.776884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.776817 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkck\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.776884 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.776844 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.777046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.777021 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.777193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.777174 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.777280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.777266 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.877986 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878189 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878065 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878272 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878411 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878487 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878499 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878431 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878584 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878602 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkck\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878634 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878856 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878748 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878916 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878888 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.878973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.878913 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.879530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.879507 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.880793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.880766 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.881011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.880990 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.899733 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.899698 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkck\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:05.900274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:05.900248 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:06.028773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.028748 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:06.346072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.346041 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft"] Oct 29 22:46:06.346988 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:46:06.346962 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6d5106_733c_4ebd_af8c_c88968b0f0cf.slice/crio-0f6e2227c45cc2262afedc308794c8a87311b8d10cca6bfde9a6962c010e6bbe WatchSource:0}: Error finding container 0f6e2227c45cc2262afedc308794c8a87311b8d10cca6bfde9a6962c010e6bbe: Status 404 returned error can't find the container with id 0f6e2227c45cc2262afedc308794c8a87311b8d10cca6bfde9a6962c010e6bbe Oct 29 22:46:06.349897 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.349847 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:46:06.349996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.349930 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:46:06.349996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.349962 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:46:06.748242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.748175 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" event={"ID":"9b6d5106-733c-4ebd-af8c-c88968b0f0cf","Type":"ContainerStarted","Data":"3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2"} Oct 29 22:46:06.748242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.748243 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" event={"ID":"9b6d5106-733c-4ebd-af8c-c88968b0f0cf","Type":"ContainerStarted","Data":"0f6e2227c45cc2262afedc308794c8a87311b8d10cca6bfde9a6962c010e6bbe"} Oct 29 22:46:06.801692 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:06.801636 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podStartSLOduration=1.801618283 podStartE2EDuration="1.801618283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:06.795746256 +0000 UTC m=+1882.111140338" watchObservedRunningTime="2026-10-29 22:46:06.801618283 +0000 UTC m=+1882.117012367" Oct 29 22:46:07.029128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:07.029045 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:07.030972 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:07.030942 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 29 22:46:07.031128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:07.031000 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 29 22:46:08.045700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:08.044317 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 29 22:46:08.045700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:08.044372 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 29 22:46:09.029917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:09.029710 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 29 22:46:09.029917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:09.029771 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 29 22:46:09.363985 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:46:09.363880 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:46:10.033586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:10.033543 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.033774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:10.033610 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 29 22:46:11.035207 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:11.029684 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 29 22:46:11.035207 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:11.029761 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 29 22:46:12.029558 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:12.029513 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.029780 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:12.029608 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 29 22:46:13.035455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:13.035422 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:13.035906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:13.035888 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:13.036635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:13.036618 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:46:20.421960 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:20.421924 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg"] Oct 29 22:46:20.422433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:20.422202 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" containerID="cri-o://f5816bb7cc48f9aa8425bad733ab23ffa828945da140238f95c54cf743f09891" gracePeriod=30 Oct 29 22:46:20.435183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:20.435137 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4"] Oct 29 22:46:20.435392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:20.435369 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" containerID="cri-o://d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b" gracePeriod=30 Oct 29 22:46:20.455647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:20.455623 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp"] Oct 29 22:46:20.455813 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:20.455795 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" containerID="cri-o://85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be" gracePeriod=30 Oct 29 22:46:25.826176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.813200 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:46:25.846833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.846802 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:46:25.921990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.921951 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922001 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922036 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922088 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.922298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922190 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922298 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922225 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922331 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922401 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.922599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922463 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922542 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.922599 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922591 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.922875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922671 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.922875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922706 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.922875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922785 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.922875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922827 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.923188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922883 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.923188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922911 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket\") pod \"6247cc5b-7cda-452f-bd91-9e22ec75955a\" (UID: \"6247cc5b-7cda-452f-bd91-9e22ec75955a\") " Oct 29 22:46:25.923188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.922952 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo\") pod \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\" (UID: \"ba90d129-8ae5-4a2d-954a-cc532543b6ac\") " Oct 29 22:46:25.924741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.924671 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.925126 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.925417 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.925612 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.925915 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.926271 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.926601 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.927134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.926880 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.929773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.929110 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.929773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.929354 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.932469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.931597 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.945280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.940077 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.945280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.940221 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:25.951260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.947805 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:25.960238 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.956835 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "kube-api-access-jwfh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.960238 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.956930 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "kube-api-access-4d78l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.966516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.963295 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy" pod "ba90d129-8ae5-4a2d-954a-cc532543b6ac" (UID: "ba90d129-8ae5-4a2d-954a-cc532543b6ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.966713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.966684 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy" pod "6247cc5b-7cda-452f-bd91-9e22ec75955a" (UID: "6247cc5b-7cda-452f-bd91-9e22ec75955a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.990896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.990861 5073 generic.go:356] "Generic (PLEG): container finished" podID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerID="f5816bb7cc48f9aa8425bad733ab23ffa828945da140238f95c54cf743f09891" exitCode=0 Oct 29 22:46:25.991210 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.990928 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" event={"ID":"b1f39109-3fc4-415b-aa44-bade3be9f985","Type":"ContainerDied","Data":"f5816bb7cc48f9aa8425bad733ab23ffa828945da140238f95c54cf743f09891"} Oct 29 22:46:26.000836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.993331 5073 generic.go:356] "Generic (PLEG): container finished" podID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerID="85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be" exitCode=0 Oct 29 22:46:26.000836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.993491 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" Oct 29 22:46:26.000836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.994225 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" event={"ID":"ba90d129-8ae5-4a2d-954a-cc532543b6ac","Type":"ContainerDied","Data":"85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be"} Oct 29 22:46:26.000836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.994267 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp" event={"ID":"ba90d129-8ae5-4a2d-954a-cc532543b6ac","Type":"ContainerDied","Data":"e522b7e97523e129ed6d8685a3ff02eb4c27eb8cba80022d7d65c7bf2bec5619"} Oct 29 22:46:26.000836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:25.994288 5073 scope.go:122] "RemoveContainer" containerID="85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be" Oct 29 22:46:26.017485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.017410 5073 generic.go:356] "Generic (PLEG): container finished" podID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerID="d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b" exitCode=0 Oct 29 22:46:26.017638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.017620 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" event={"ID":"6247cc5b-7cda-452f-bd91-9e22ec75955a","Type":"ContainerDied","Data":"d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b"} Oct 29 22:46:26.017720 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.017685 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" Oct 29 22:46:26.017779 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.017721 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4" event={"ID":"6247cc5b-7cda-452f-bd91-9e22ec75955a","Type":"ContainerDied","Data":"df1eaa0bd0b88eb273544ee1f351c9473d4fb753a57361e1c9b40744c5055db2"} Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028246 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028280 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028294 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028306 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028319 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028335 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028348 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028363 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028398 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028415 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfh4\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-kube-api-access-jwfh4\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028430 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/ba90d129-8ae5-4a2d-954a-cc532543b6ac-kube-api-access-4d78l\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028445 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028461 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba90d129-8ae5-4a2d-954a-cc532543b6ac-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028474 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028486 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba90d129-8ae5-4a2d-954a-cc532543b6ac-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028498 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028516 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6247cc5b-7cda-452f-bd91-9e22ec75955a-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.033434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.028528 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6247cc5b-7cda-452f-bd91-9e22ec75955a-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.039689 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.039663 5073 scope.go:122] "RemoveContainer" containerID="85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be" Oct 29 22:46:26.040084 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:46:26.040058 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be\": container with ID starting with 85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be not found: ID does not exist" containerID="85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be" Oct 29 22:46:26.040125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.040099 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be"} err="failed to get container status \"85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be\": rpc error: code = NotFound desc = could not find container \"85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be\": container with ID starting with 85ddb69106afccf71d13ae39e8c99c784a34a23cd128edda0bac26fda91de6be not found: ID does not exist" Oct 29 22:46:26.040201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.040122 5073 scope.go:122] "RemoveContainer" containerID="d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b" Oct 29 22:46:26.059112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.059088 5073 scope.go:122] "RemoveContainer" containerID="d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b" Oct 29 22:46:26.059483 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:46:26.059458 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b\": container with ID starting with d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b not found: ID does not exist" containerID="d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b" Oct 29 22:46:26.059567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.059495 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b"} err="failed to get container status \"d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b\": rpc error: code = NotFound desc = could not find container \"d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b\": container with ID starting with d75f964497639a021a3301dd7f6d73455aaa3d16c8fe893b451d19caab6f610b not found: ID does not exist" Oct 29 22:46:26.080932 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.080669 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp"] Oct 29 22:46:26.084676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.084656 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-84kjp"] Oct 29 22:46:26.122308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.122281 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4"] Oct 29 22:46:26.127283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.127256 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-p5nx4"] Oct 29 22:46:26.180416 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.180389 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:46:26.336205 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.336039 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.336976 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337200 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337180 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337259 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337218 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337277 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337349 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337389 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337450 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.337626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.337605 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy\") pod \"b1f39109-3fc4-415b-aa44-bade3be9f985\" (UID: \"b1f39109-3fc4-415b-aa44-bade3be9f985\") " Oct 29 22:46:26.339078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.338672 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.339078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.339024 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.339078 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.339039 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.339323 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.339108 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.339323 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.339290 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.341266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.341238 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.341400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.341375 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.343318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.342601 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "kube-api-access-5nd6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.343318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.343060 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy" pod "b1f39109-3fc4-415b-aa44-bade3be9f985" (UID: "b1f39109-3fc4-415b-aa44-bade3be9f985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.439674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439645 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f39109-3fc4-415b-aa44-bade3be9f985-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439670 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439679 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439687 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nd6q\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-kube-api-access-5nd6q\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439697 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439706 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439714 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439722 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f39109-3fc4-415b-aa44-bade3be9f985-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.439896 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.439730 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f39109-3fc4-415b-aa44-bade3be9f985-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.844390 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.844346 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" path="/var/lib/kubelet/pods/6247cc5b-7cda-452f-bd91-9e22ec75955a/volumes" Oct 29 22:46:26.845264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:26.845235 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" path="/var/lib/kubelet/pods/ba90d129-8ae5-4a2d-954a-cc532543b6ac/volumes" Oct 29 22:46:27.025938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:27.025907 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" Oct 29 22:46:27.025938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:27.025924 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg" event={"ID":"b1f39109-3fc4-415b-aa44-bade3be9f985","Type":"ContainerDied","Data":"5d9874ceab7ac7d0fbdd2f5d7d63da3caaeda4b35e2dabc1455f5cb22e6e5d10"} Oct 29 22:46:27.026253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:27.025988 5073 scope.go:122] "RemoveContainer" containerID="f5816bb7cc48f9aa8425bad733ab23ffa828945da140238f95c54cf743f09891" Oct 29 22:46:27.084652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:27.084629 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg"] Oct 29 22:46:27.097788 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:27.097729 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-6zczg"] Oct 29 22:46:28.864371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:28.862543 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" path="/var/lib/kubelet/pods/b1f39109-3fc4-415b-aa44-bade3be9f985/volumes" Oct 29 22:46:39.257672 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:46:39.257635 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:46:46.025102 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:46.025070 5073 scope.go:122] "RemoveContainer" containerID="fdd76d093e58e6344c3b752624bcebba9cc23042582b163dad406e855942dfd6" Oct 29 22:46:50.848363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.848325 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd"] Oct 29 22:46:50.849075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849047 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" Oct 29 22:46:50.849075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849070 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849092 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849097 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849108 5073 cpu_manager.go:414] "containerMap: removing container" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849114 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849125 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849129 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7029b8b-85ef-4cde-8e00-f592b71056a0" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849139 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849144 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849181 5073 cpu_manager.go:414] "containerMap: removing container" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849186 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb6e0a-9fd5-4ba7-ba3a-08cd48f3e9c6" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849194 5073 cpu_manager.go:414] "containerMap: removing container" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849200 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594df621-447e-4232-ab84-67bad2b0b940" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849208 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849212 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849226 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" Oct 29 22:46:50.849266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849230 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849326 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8076aef0-abbb-4aba-9d5f-6afcabe62e28" containerName="cleanup" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849335 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1aed31-3ae5-409e-a08c-2f1804124a3b" containerName="cleanup" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849344 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d6c86d-53c7-47ce-9597-d84fac5f3ae8" containerName="cleanup" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849351 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ccf14b-4022-4e3a-9757-458e77b0c827" containerName="cleanup" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849362 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba90d129-8ae5-4a2d-954a-cc532543b6ac" containerName="istio-proxy" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849369 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6247cc5b-7cda-452f-bd91-9e22ec75955a" containerName="istio-proxy" Oct 29 22:46:50.849947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.849377 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f39109-3fc4-415b-aa44-bade3be9f985" containerName="istio-proxy" Oct 29 22:46:50.854931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.854905 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.882073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.882041 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd"] Oct 29 22:46:50.929584 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.929534 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.929764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.929651 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.929764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.929725 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.929891 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.929810 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.929951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.929932 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wst\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.930010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.929984 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.930093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.930078 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.930216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.930201 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:50.930318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:50.930295 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032261 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032306 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032418 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032444 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032480 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032520 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032574 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wst\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032610 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032641 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032697 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032726 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.032954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.032791 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.033129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.033027 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.033129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.033084 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.035204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.035145 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.035319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.035264 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.058220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.058188 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wst\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.060873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.060848 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.173695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.173605 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:51.503843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.503794 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd"] Oct 29 22:46:51.504292 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:46:51.504261 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f28fc48_9277_42f8_90d7_b1ef9d6a3b86.slice/crio-7bf9ab5e32e147553aa286a5b221ddbaf1e33432a1211d1b3198c3be80f3b85e WatchSource:0}: Error finding container 7bf9ab5e32e147553aa286a5b221ddbaf1e33432a1211d1b3198c3be80f3b85e: Status 404 returned error can't find the container with id 7bf9ab5e32e147553aa286a5b221ddbaf1e33432a1211d1b3198c3be80f3b85e Oct 29 22:46:51.507227 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.507191 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:46:51.507323 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.507269 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:46:51.507323 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.507299 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:46:51.568678 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.568640 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85"] Oct 29 22:46:51.568854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:51.568831 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" containerID="cri-o://c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51" gracePeriod=30 Oct 29 22:46:52.400617 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:52.400578 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" event={"ID":"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86","Type":"ContainerStarted","Data":"2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8"} Oct 29 22:46:52.401048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:52.400647 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" event={"ID":"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86","Type":"ContainerStarted","Data":"7bf9ab5e32e147553aa286a5b221ddbaf1e33432a1211d1b3198c3be80f3b85e"} Oct 29 22:46:52.467996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:52.467934 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podStartSLOduration=2.467918413 podStartE2EDuration="2.467918413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:52.466785713 +0000 UTC m=+1927.782179794" watchObservedRunningTime="2026-10-29 22:46:52.467918413 +0000 UTC m=+1927.783312492" Oct 29 22:46:53.174778 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:53.174745 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:53.176886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:53.176858 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:46:53.177030 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:53.176914 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 29 22:46:54.174236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:54.174139 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:46:54.174236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:54.174210 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 29 22:46:55.244175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:55.243775 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:46:55.244175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:55.243837 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 29 22:46:55.845436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:55.845396 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:55.845639 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:55.845452 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:56.204239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:56.203952 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:46:56.204239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:56.204039 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 29 22:46:57.147671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.147627 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:46:57.174719 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.174686 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:46:57.174869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.174760 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 29 22:46:57.277595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.277511 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.277595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.277578 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.277820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.277614 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.277820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.277687 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.277941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.277835 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.278362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.278320 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.278566 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.277851 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.278825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.278793 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.278967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.278855 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.279316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.279008 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.279316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.279112 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token\") pod \"d216984e-69e8-445d-b3e0-334ca271c0fd\" (UID: \"d216984e-69e8-445d-b3e0-334ca271c0fd\") " Oct 29 22:46:57.280080 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.279682 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:57.280080 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.279990 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.280080 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.280029 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.280469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.280449 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d216984e-69e8-445d-b3e0-334ca271c0fd-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.280547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.280473 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.280547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.280486 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.280547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.280500 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.280547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.280514 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.281350 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.281314 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "kube-api-access-zvp45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.281459 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.281355 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.282723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.282690 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.283434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.283411 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo" pod "d216984e-69e8-445d-b3e0-334ca271c0fd" (UID: "d216984e-69e8-445d-b3e0-334ca271c0fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:57.381759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.381723 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.381759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.381753 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.381759 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.381762 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d216984e-69e8-445d-b3e0-334ca271c0fd-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.382017 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.381772 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvp45\" (UniqueName: \"kubernetes.io/projected/d216984e-69e8-445d-b3e0-334ca271c0fd-kube-api-access-zvp45\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.511171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.504054 5073 generic.go:356] "Generic (PLEG): container finished" podID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerID="c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51" exitCode=0 Oct 29 22:46:57.511171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.504124 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" event={"ID":"d216984e-69e8-445d-b3e0-334ca271c0fd","Type":"ContainerDied","Data":"c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51"} Oct 29 22:46:57.511171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.504194 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" event={"ID":"d216984e-69e8-445d-b3e0-334ca271c0fd","Type":"ContainerDied","Data":"1130e01db25c4b6bc5ab51d47d072f16b60594ed9a88c972ac213b53a6f55349"} Oct 29 22:46:57.511171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.504222 5073 scope.go:122] "RemoveContainer" containerID="c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51" Oct 29 22:46:57.511171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.504379 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85" Oct 29 22:46:57.591525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.591475 5073 scope.go:122] "RemoveContainer" containerID="c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51" Oct 29 22:46:57.592104 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:46:57.591931 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51\": container with ID starting with c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51 not found: ID does not exist" containerID="c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51" Oct 29 22:46:57.592104 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.591970 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51"} err="failed to get container status \"c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51\": rpc error: code = NotFound desc = could not find container \"c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51\": container with ID starting with c8dfdd18115e3dea9ecaafc73b8f15ab62bfa10f4f6a9061b9ea94277ad82e51 not found: ID does not exist" Oct 29 22:46:57.648185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.644124 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85"] Oct 29 22:46:57.666288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:57.658486 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-bkz85"] Oct 29 22:46:58.174773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:58.174733 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:46:58.175214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:58.174801 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 29 22:46:58.839179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:58.839133 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" path="/var/lib/kubelet/pods/d216984e-69e8-445d-b3e0-334ca271c0fd/volumes" Oct 29 22:46:59.029750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.029719 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8"] Oct 29 22:46:59.030425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.030410 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" Oct 29 22:46:59.030479 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.030426 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" Oct 29 22:46:59.030539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.030530 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d216984e-69e8-445d-b3e0-334ca271c0fd" containerName="istio-proxy" Oct 29 22:46:59.035928 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.035795 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.056092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.054299 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8"] Oct 29 22:46:59.056569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.056537 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"openshift-service-ca.crt\"" Oct 29 22:46:59.062966 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.062927 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"kube-root-ca.crt\"" Oct 29 22:46:59.063094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.063028 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"default-dockercfg-qfqv6\"" Oct 29 22:46:59.063214 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.063138 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"auth-test-model-afdbe8-kserve-self-signed-certs\"" Oct 29 22:46:59.113587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.113493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.113587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.113564 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.113793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.113711 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.113793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.113743 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.113793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.113772 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.113923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.113883 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlhp\" (UniqueName: \"kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.180342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.180311 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:59.180820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.180729 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:59.182108 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.182082 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:46:59.215296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215250 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.215496 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215355 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.215661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215636 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.215741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215683 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.215741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215719 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.215850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215727 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.215850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.215825 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlhp\" (UniqueName: \"kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.216321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.216142 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.216578 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.216429 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.218723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.218699 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.218816 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.218721 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.237125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.237087 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlhp\" (UniqueName: \"kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp\") pod \"auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.371231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.368121 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:46:59.843751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:46:59.843704 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8"] Oct 29 22:46:59.843992 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:46:59.843965 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf93e8566_49c7_4e28_9656_3ec25438df7a.slice/crio-96b20e8786d5871ca7a72c8cd7d4ad2e78c3805bfdb339b2a8cbd9b8f292c4de WatchSource:0}: Error finding container 96b20e8786d5871ca7a72c8cd7d4ad2e78c3805bfdb339b2a8cbd9b8f292c4de: Status 404 returned error can't find the container with id 96b20e8786d5871ca7a72c8cd7d4ad2e78c3805bfdb339b2a8cbd9b8f292c4de Oct 29 22:47:00.536264 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:00.536213 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" event={"ID":"f93e8566-49c7-4e28-9656-3ec25438df7a","Type":"ContainerStarted","Data":"a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725"} Oct 29 22:47:00.536786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:00.536273 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" event={"ID":"f93e8566-49c7-4e28-9656-3ec25438df7a","Type":"ContainerStarted","Data":"96b20e8786d5871ca7a72c8cd7d4ad2e78c3805bfdb339b2a8cbd9b8f292c4de"} Oct 29 22:47:00.578653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:00.578599 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" podStartSLOduration=1.5785831620000002 podStartE2EDuration="1.578583162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:00.577922716 +0000 UTC m=+1935.893316795" watchObservedRunningTime="2026-10-29 22:47:00.578583162 +0000 UTC m=+1935.893977241" Oct 29 22:47:01.368829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:01.368795 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:47:05.865355 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.865321 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54"] Oct 29 22:47:05.869603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.869580 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.893613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.893583 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54"] Oct 29 22:47:05.971851 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.971817 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbb9\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.971864 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.971883 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972006 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.971993 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.972108 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.972225 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.972257 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.972275 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:05.972349 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:05.972300 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073039 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbb9\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073084 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073101 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073141 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073256 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073311 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073337 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073361 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073399 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073693 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073621 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073764 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073803 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.073888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073854 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.074018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.073995 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.075811 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.075789 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.076044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.076027 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.093492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.093462 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.093587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.093567 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbb9\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.185112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.185055 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:06.505274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.505244 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54"] Oct 29 22:47:06.505589 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:47:06.505569 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41cca39_790e_4f63_827b_c3cd3be0fd37.slice/crio-f2d2b860a84e7f5df96978b7f6c1aa43ac851a4b6f575423d72c8ac239c9ef2d WatchSource:0}: Error finding container f2d2b860a84e7f5df96978b7f6c1aa43ac851a4b6f575423d72c8ac239c9ef2d: Status 404 returned error can't find the container with id f2d2b860a84e7f5df96978b7f6c1aa43ac851a4b6f575423d72c8ac239c9ef2d Oct 29 22:47:06.508369 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.508334 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:47:06.508448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.508400 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:47:06.508448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.508429 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:47:06.580050 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:06.580023 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" event={"ID":"d41cca39-790e-4f63-827b-c3cd3be0fd37","Type":"ContainerStarted","Data":"f2d2b860a84e7f5df96978b7f6c1aa43ac851a4b6f575423d72c8ac239c9ef2d"} Oct 29 22:47:07.589234 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:07.589191 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" event={"ID":"d41cca39-790e-4f63-827b-c3cd3be0fd37","Type":"ContainerStarted","Data":"d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c"} Oct 29 22:47:07.654842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:07.654789 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podStartSLOduration=2.654773683 podStartE2EDuration="2.654773683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:07.653018364 +0000 UTC m=+1942.968412473" watchObservedRunningTime="2026-10-29 22:47:07.654773683 +0000 UTC m=+1942.970167763" Oct 29 22:47:08.185588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:08.185554 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:08.187565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:08.187539 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:47:08.187721 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:08.187591 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 22:47:09.185651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:09.185617 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:47:09.186034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:09.185681 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 22:47:09.261347 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:47:09.261312 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:47:09.369340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:09.369306 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:47:09.384582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:09.384539 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:47:09.647387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:09.647362 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:47:10.185560 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:10.185529 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:47:10.185812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:10.185586 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 22:47:11.186039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:11.186000 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:47:11.186546 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:11.186066 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 22:47:12.044192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:12.044135 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt"] Oct 29 22:47:12.053924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:12.053891 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73734f08-f29qt"] Oct 29 22:47:12.185913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:12.185854 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:47:12.186249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:12.186214 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 22:47:12.871201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:12.870227 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35659891-4589-473e-b226-ba3fe55c328f" path="/var/lib/kubelet/pods/35659891-4589-473e-b226-ba3fe55c328f/volumes" Oct 29 22:47:13.190314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:13.190235 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:13.190703 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:13.190674 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:13.191420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:13.191397 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:47:24.396629 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:24.396587 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:47:39.266144 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:47:39.266103 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:47:51.506540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.506503 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22"] Oct 29 22:47:51.513625 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.513599 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.536689 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.536663 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22"] Oct 29 22:47:51.546219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.546192 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"openshift-service-ca.crt\"" Oct 29 22:47:51.546371 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.546228 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"kube-root-ca.crt\"" Oct 29 22:47:51.550494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.550472 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"auth-test-model-84304f-kserve-self-signed-certs\"" Oct 29 22:47:51.550494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.550472 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"default-dockercfg-n2599\"" Oct 29 22:47:51.616537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.616504 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.617062 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.617037 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.617241 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.617213 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.617290 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.617261 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvx7\" (UniqueName: \"kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.617366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.617346 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.617403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.617382 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.718593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718558 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.718781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718634 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.718781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718659 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvx7\" (UniqueName: \"kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.718781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718699 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.718781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718779 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.719018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718897 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.719018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.718998 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.719126 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.719106 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.719379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.719341 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.721265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.721239 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.722574 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.722549 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.797225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.780071 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvx7\" (UniqueName: \"kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7\") pod \"auth-test-model-84304f-kserve-84f5444469-tvp22\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:51.835026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:51.834989 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:52.250063 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:52.250035 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22"] Oct 29 22:47:52.251591 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:47:52.251566 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5391ed24_156b_4131_9964_1c8ab042cc2c.slice/crio-bacb004449f4a336521065edd6046772060ae4ff0dd83f24ea7227b1c4d4e787 WatchSource:0}: Error finding container bacb004449f4a336521065edd6046772060ae4ff0dd83f24ea7227b1c4d4e787: Status 404 returned error can't find the container with id bacb004449f4a336521065edd6046772060ae4ff0dd83f24ea7227b1c4d4e787 Oct 29 22:47:53.172255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:53.172205 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" event={"ID":"5391ed24-156b-4131-9964-1c8ab042cc2c","Type":"ContainerStarted","Data":"5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454"} Oct 29 22:47:53.172676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:53.172259 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" event={"ID":"5391ed24-156b-4131-9964-1c8ab042cc2c","Type":"ContainerStarted","Data":"bacb004449f4a336521065edd6046772060ae4ff0dd83f24ea7227b1c4d4e787"} Oct 29 22:47:53.247709 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:53.247662 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" podStartSLOduration=2.247648119 podStartE2EDuration="2.247648119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:53.243292111 +0000 UTC m=+1988.558686190" watchObservedRunningTime="2026-10-29 22:47:53.247648119 +0000 UTC m=+1988.563042199" Oct 29 22:47:53.836720 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:53.836689 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:47:58.848295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.848263 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5"] Oct 29 22:47:58.852480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.852462 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.870347 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.870312 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-afdbe8-openshift-default-dockercfg-crwqv\"" Oct 29 22:47:58.872251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872224 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872304 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872362 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872486 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872408 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872537 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872580 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872640 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872829 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.872926 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.872909 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7fn\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.887088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.887059 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5"] Oct 29 22:47:58.974273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974245 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974282 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974309 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974413 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974478 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7fn\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974536 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974724 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974760 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974769 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974842 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.974919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.974880 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.975184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.975038 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.975184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.975048 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.975322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.975213 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.977198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.977174 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.977432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.977415 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.997431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.997403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7fn\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:58.999231 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:58.999213 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token\") pod \"e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:59.168664 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:59.168595 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:47:59.706071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:59.706041 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5"] Oct 29 22:47:59.706256 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:47:59.706126 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66145ed4_2278_4025_ac12_639607aeae8f.slice/crio-38b4231fcbde78e5a19039c8bcd8a9b2a0a6e4786613abc988759f863ccf78c8 WatchSource:0}: Error finding container 38b4231fcbde78e5a19039c8bcd8a9b2a0a6e4786613abc988759f863ccf78c8: Status 404 returned error can't find the container with id 38b4231fcbde78e5a19039c8bcd8a9b2a0a6e4786613abc988759f863ccf78c8 Oct 29 22:47:59.709009 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:59.708975 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:47:59.709101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:59.709048 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:47:59.709101 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:47:59.709079 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:48:00.243797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:00.243744 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" event={"ID":"66145ed4-2278-4025-ac12-639607aeae8f","Type":"ContainerStarted","Data":"68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356"} Oct 29 22:48:00.244557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:00.244511 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" event={"ID":"66145ed4-2278-4025-ac12-639607aeae8f","Type":"ContainerStarted","Data":"38b4231fcbde78e5a19039c8bcd8a9b2a0a6e4786613abc988759f863ccf78c8"} Oct 29 22:48:00.305204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:00.305129 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podStartSLOduration=2.305114194 podStartE2EDuration="2.305114194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:00.303491362 +0000 UTC m=+1995.618885455" watchObservedRunningTime="2026-10-29 22:48:00.305114194 +0000 UTC m=+1995.620508274" Oct 29 22:48:01.169031 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:01.168991 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:48:01.170270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:01.170243 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:48:01.170441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:01.170294 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 22:48:01.842325 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:01.841362 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:48:02.002204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:01.990538 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:48:02.174233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:02.173833 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:48:02.174233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:02.173904 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 22:48:02.346175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:02.346018 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:48:03.169500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:03.169462 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:48:03.169918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:03.169526 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 22:48:04.169305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:04.169268 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:48:04.169474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:04.169333 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 22:48:05.169704 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:05.169667 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:48:05.170275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:05.169732 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 22:48:06.184451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:06.183518 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:48:06.184451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:06.183578 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 22:48:07.175094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:07.175057 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:48:07.175503 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:07.175475 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:48:07.176217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:07.176195 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 22:48:09.257255 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:48:09.257219 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:48:16.569511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.569476 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h"] Oct 29 22:48:16.574299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.574278 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.597000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.596976 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-dbb507-serving-cert\"" Oct 29 22:48:16.613900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.613798 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h"] Oct 29 22:48:16.694411 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.694376 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.694627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.694601 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.694732 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.694704 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.694949 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.694933 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4z6l\" (UniqueName: \"kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.799324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.799279 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4z6l\" (UniqueName: \"kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.822426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.822334 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.822686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.822662 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.822914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.822895 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.824610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.824584 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.852935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.852904 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.865570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.865502 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.877044 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.877015 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4z6l\" (UniqueName: \"kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l\") pod \"maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:16.892004 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:16.891968 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:17.225551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.225524 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h"] Oct 29 22:48:17.228869 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:48:17.228841 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c13e2c9_7328_460a_87b1_444a9e8be8be.slice/crio-77a15dc13edc0466e5cf60cae0d375253181a706e053a5b627d74e0b4f949b8d WatchSource:0}: Error finding container 77a15dc13edc0466e5cf60cae0d375253181a706e053a5b627d74e0b4f949b8d: Status 404 returned error can't find the container with id 77a15dc13edc0466e5cf60cae0d375253181a706e053a5b627d74e0b4f949b8d Oct 29 22:48:17.230699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.230674 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:48:17.392107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.392064 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz"] Oct 29 22:48:17.398739 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.398715 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.415546 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.415518 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-172c10\"/\"default-dockercfg-jnxhf\"" Oct 29 22:48:17.415676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.415659 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-172c10\"/\"openshift-service-ca.crt\"" Oct 29 22:48:17.422835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.422495 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz"] Oct 29 22:48:17.426510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.426353 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-172c10\"/\"test-model-172c10-kserve-self-signed-certs\"" Oct 29 22:48:17.426510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.426457 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-172c10\"/\"kube-root-ca.crt\"" Oct 29 22:48:17.432793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.432758 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.432913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.432858 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.433028 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.433010 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.433111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.433043 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.433236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.433122 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.433236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.433193 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gkl\" (UniqueName: \"kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.442425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.442391 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" event={"ID":"4c13e2c9-7328-460a-87b1-444a9e8be8be","Type":"ContainerStarted","Data":"77a15dc13edc0466e5cf60cae0d375253181a706e053a5b627d74e0b4f949b8d"} Oct 29 22:48:17.535098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535012 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535266 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535238 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535273 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535349 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535411 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gkl\" (UniqueName: \"kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535453 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535498 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535518 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.535847 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.535820 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.537934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.537897 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.538095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.538080 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.557374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.557354 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gkl\" (UniqueName: \"kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl\") pod \"test-model-172c10-kserve-68f84c94c9-w27sz\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:17.734270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:17.727183 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:18.364592 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:18.364235 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz"] Oct 29 22:48:18.365720 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:48:18.365692 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod097780fe_e0f8_408a_bd92_b8bf41d881ad.slice/crio-60f193628c61f43fcd4091f749b92d606cc3aa56e20df6f1dbaae430257fec3c WatchSource:0}: Error finding container 60f193628c61f43fcd4091f749b92d606cc3aa56e20df6f1dbaae430257fec3c: Status 404 returned error can't find the container with id 60f193628c61f43fcd4091f749b92d606cc3aa56e20df6f1dbaae430257fec3c Oct 29 22:48:18.459933 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:18.459901 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" event={"ID":"097780fe-e0f8-408a-bd92-b8bf41d881ad","Type":"ContainerStarted","Data":"60f193628c61f43fcd4091f749b92d606cc3aa56e20df6f1dbaae430257fec3c"} Oct 29 22:48:18.461761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:18.461732 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" event={"ID":"4c13e2c9-7328-460a-87b1-444a9e8be8be","Type":"ContainerStarted","Data":"ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60"} Oct 29 22:48:18.461924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:18.461905 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:18.522328 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:18.522283 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" podStartSLOduration=2.143325748 podStartE2EDuration="2.522268303s" totalImagesPullingTime="378.942555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:18.52076378 +0000 UTC m=+2013.836157860" watchObservedRunningTime="2026-10-29 22:48:18.522268303 +0000 UTC m=+2013.837662383" Oct 29 22:48:19.470997 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:19.470955 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" event={"ID":"097780fe-e0f8-408a-bd92-b8bf41d881ad","Type":"ContainerStarted","Data":"95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb"} Oct 29 22:48:19.523311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:19.523265 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" podStartSLOduration=2.523251885 podStartE2EDuration="2.523251885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:19.51924028 +0000 UTC m=+2014.834634365" watchObservedRunningTime="2026-10-29 22:48:19.523251885 +0000 UTC m=+2014.838645965" Oct 29 22:48:19.728654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:19.728537 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:24.477414 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:24.477330 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 22:48:25.537677 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:25.537602 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk"] Oct 29 22:48:27.728212 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:27.728181 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:27.740905 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:27.740881 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:28.563521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:28.563494 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:48:39.257094 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:48:39.257044 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:48:40.598867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.598817 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" podUID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" containerName="main" containerID="cri-o://91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082" gracePeriod=45 Oct 29 22:48:40.903011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.902970 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:48:40.953091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953063 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache\") pod \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " Oct 29 22:48:40.953253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953169 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home\") pod \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " Oct 29 22:48:40.953317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953267 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs\" (UniqueName: \"kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs\") pod \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " Oct 29 22:48:40.953372 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953312 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir\") pod \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " Oct 29 22:48:40.953426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953358 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache" pod "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" (UID: "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:40.953502 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953474 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home" pod "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" (UID: "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:40.953570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953522 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x\" (UniqueName: \"kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x\") pod \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " Oct 29 22:48:40.953662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953648 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm\") pod \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\" (UID: \"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516\") " Oct 29 22:48:40.953736 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.953683 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir" pod "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" (UID: "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:40.955261 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.955235 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.955397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.955263 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.955397 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.955277 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.956638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.956613 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs" pod "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" (UID: "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:40.956865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.956842 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x" pod "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" (UID: "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516"). InnerVolumeSpecName "kube-api-access-xpz8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:40.957505 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:40.957482 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm" pod "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" (UID: "d97b71c3-05d8-4f9d-b3d4-ff81aacd5516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:41.057131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.057103 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpz8x\" (UniqueName: \"kubernetes.io/projected/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-kube-api-access-xpz8x\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:48:41.057131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.057131 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:48:41.057286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.057140 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:48:41.649193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.649140 5073 generic.go:356] "Generic (PLEG): container finished" podID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" containerID="91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082" exitCode=0 Oct 29 22:48:41.649619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.649229 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" Oct 29 22:48:41.649619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.649234 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" event={"ID":"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516","Type":"ContainerDied","Data":"91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082"} Oct 29 22:48:41.649619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.649284 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" event={"ID":"d97b71c3-05d8-4f9d-b3d4-ff81aacd5516","Type":"ContainerDied","Data":"66a5d7debafa7570465c21444dab48740144d10f49abd4e178bac5aa95b2a61c"} Oct 29 22:48:41.649619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.649302 5073 scope.go:122] "RemoveContainer" containerID="91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082" Oct 29 22:48:41.661090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.661074 5073 scope.go:122] "RemoveContainer" containerID="91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082" Oct 29 22:48:41.661382 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:48:41.661363 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082\": container with ID starting with 91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082 not found: ID does not exist" containerID="91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082" Oct 29 22:48:41.661448 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.661391 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082"} err="failed to get container status \"91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082\": rpc error: code = NotFound desc = could not find container \"91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082\": container with ID starting with 91747b501f30a5c8abb1e317e215608431153bdc4a7f4d0720e7de69a6db8082 not found: ID does not exist" Oct 29 22:48:41.712276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.712230 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk"] Oct 29 22:48:41.729305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.728902 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk"] Oct 29 22:48:41.786688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:41.786650 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-139eb7/test-model-139eb7-kserve-68dbc897b5-cklxk" podUID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" containerName="main" probeResult="failure" output="Get \"https://10.132.0.137:8000/ready\": context deadline exceeded" Oct 29 22:48:42.840763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:42.840728 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" path="/var/lib/kubelet/pods/d97b71c3-05d8-4f9d-b3d4-ff81aacd5516/volumes" Oct 29 22:48:46.137269 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:46.137237 5073 scope.go:122] "RemoveContainer" containerID="06fca2069b3e14c6a7eae225070cb75a4be9fe47ff7635a397972f5e2683c215" Oct 29 22:48:48.087787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:48:48.087747 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:49:04.858880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.858838 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp"] Oct 29 22:49:04.860547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.860518 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" containerName="main" Oct 29 22:49:04.860547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.860550 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" containerName="main" Oct 29 22:49:04.861187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.861166 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97b71c3-05d8-4f9d-b3d4-ff81aacd5516" containerName="main" Oct 29 22:49:04.868643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.868616 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.902275 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.902247 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-379a6d-openshift-default-dockercfg-zn4lk\"" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.910709 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.910771 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.910799 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.910859 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtwv\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.910980 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.911054 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.911101 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.911253 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.916630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.911325 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:04.918354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:04.917759 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp"] Oct 29 22:49:05.013056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013019 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013138 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013191 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013214 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013267 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtwv\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013378 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013454 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013529 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013503 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013691 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013571 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013691 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013635 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.013855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.013835 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.014198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.014127 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.014443 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.014418 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.014525 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.014442 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.016627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.016601 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.016722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.016640 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.034553 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.034524 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.034740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.034722 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtwv\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.233833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.233800 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:05.564285 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.564256 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp"] Oct 29 22:49:05.564895 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:49:05.564866 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9330a61_e457_47d4_bbbb_110325deecc3.slice/crio-c39dcfe7fbc7249566fcd7d5908c4ea5380538cc8d2793327d287e84af5a2762 WatchSource:0}: Error finding container c39dcfe7fbc7249566fcd7d5908c4ea5380538cc8d2793327d287e84af5a2762: Status 404 returned error can't find the container with id c39dcfe7fbc7249566fcd7d5908c4ea5380538cc8d2793327d287e84af5a2762 Oct 29 22:49:05.568192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.568131 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:49:05.568287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.568239 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:49:05.568287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.568280 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:49:05.967226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.967186 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" event={"ID":"e9330a61-e457-47d4-bbbb-110325deecc3","Type":"ContainerStarted","Data":"98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232"} Oct 29 22:49:05.967748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:05.967254 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" event={"ID":"e9330a61-e457-47d4-bbbb-110325deecc3","Type":"ContainerStarted","Data":"c39dcfe7fbc7249566fcd7d5908c4ea5380538cc8d2793327d287e84af5a2762"} Oct 29 22:49:06.035202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:06.035116 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podStartSLOduration=2.035098367 podStartE2EDuration="2.035098367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:06.031409892 +0000 UTC m=+2061.346803972" watchObservedRunningTime="2026-10-29 22:49:06.035098367 +0000 UTC m=+2061.350492447" Oct 29 22:49:06.234494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:06.234405 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:06.236674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:06.236639 5073 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:49:06.236802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:06.236716 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 22:49:07.234723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:07.234690 5073 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:49:07.235203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:07.234742 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 22:49:08.236295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:08.236245 5073 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:49:08.236792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:08.236309 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 22:49:09.234695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:09.234656 5073 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:49:09.234913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:09.234727 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 22:49:09.262303 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:49:09.262269 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:49:10.236584 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:10.236545 5073 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:49:10.236906 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:10.236864 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 22:49:11.234650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:11.234611 5073 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:49:11.235160 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:11.234672 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 22:49:12.240728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:12.240698 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:12.241224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:12.241024 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:12.241771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:12.241754 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:49:24.323434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:24.323398 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5"] Oct 29 22:49:24.323960 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:24.323682 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" containerID="cri-o://3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05" gracePeriod=30 Oct 29 22:49:29.628441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.628057 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:49:29.729667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.729626 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.729877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.729691 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.729877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.729727 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.729877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.729770 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.729877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.729844 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.729992 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730032 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730067 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730139 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730223 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert\") pod \"790bb1ce-002a-4745-b825-467d8872aaec\" (UID: \"790bb1ce-002a-4745-b825-467d8872aaec\") " Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730306 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730642 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.730821 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730784 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.731332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.730886 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:29.731332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.731115 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.731332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.731136 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.731332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.731170 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bb1ce-002a-4745-b825-467d8872aaec-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.731332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.731184 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.731332 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.731197 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.732678 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.732618 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:49:29.734099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.734072 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.735695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.735499 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "kube-api-access-pnd8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:29.735695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.735603 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token" pod "790bb1ce-002a-4745-b825-467d8872aaec" (UID: "790bb1ce-002a-4745-b825-467d8872aaec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:29.832071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.832037 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.832071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.832069 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bb1ce-002a-4745-b825-467d8872aaec-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.832329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.832081 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnd8c\" (UniqueName: \"kubernetes.io/projected/790bb1ce-002a-4745-b825-467d8872aaec-kube-api-access-pnd8c\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.832329 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:29.832095 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bb1ce-002a-4745-b825-467d8872aaec-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:49:30.223016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.222977 5073 generic.go:356] "Generic (PLEG): container finished" podID="790bb1ce-002a-4745-b825-467d8872aaec" containerID="3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05" exitCode=0 Oct 29 22:49:30.223244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.223053 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" event={"ID":"790bb1ce-002a-4745-b825-467d8872aaec","Type":"ContainerDied","Data":"3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05"} Oct 29 22:49:30.223244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.223067 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" Oct 29 22:49:30.223244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.223103 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5" event={"ID":"790bb1ce-002a-4745-b825-467d8872aaec","Type":"ContainerDied","Data":"c3ed213a07ab21e512bb74c668d55f57d269d06975c9859fced364e86f6f8a0e"} Oct 29 22:49:30.223244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.223130 5073 scope.go:122] "RemoveContainer" containerID="3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05" Oct 29 22:49:30.236733 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.236711 5073 scope.go:122] "RemoveContainer" containerID="a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f" Oct 29 22:49:30.246825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.246803 5073 scope.go:122] "RemoveContainer" containerID="3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05" Oct 29 22:49:30.247129 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:49:30.247107 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05\": container with ID starting with 3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05 not found: ID does not exist" containerID="3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05" Oct 29 22:49:30.247273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.247141 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05"} err="failed to get container status \"3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05\": rpc error: code = NotFound desc = could not find container \"3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05\": container with ID starting with 3e167aa04ce5f03f8e4b65efd640e52253c6fcff5764b1d7ede9e77704487b05 not found: ID does not exist" Oct 29 22:49:30.247273 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.247234 5073 scope.go:122] "RemoveContainer" containerID="a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f" Oct 29 22:49:30.247526 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:49:30.247507 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f\": container with ID starting with a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f not found: ID does not exist" containerID="a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f" Oct 29 22:49:30.247567 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.247531 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f"} err="failed to get container status \"a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f\": rpc error: code = NotFound desc = could not find container \"a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f\": container with ID starting with a494d0599c31e216dac0b15eea19661461bffbdb57592bfabab3eed2a8bdaa9f not found: ID does not exist" Oct 29 22:49:30.281218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.281184 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5"] Oct 29 22:49:30.294893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.294856 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-nllf5"] Oct 29 22:49:30.841218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:30.841161 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790bb1ce-002a-4745-b825-467d8872aaec" path="/var/lib/kubelet/pods/790bb1ce-002a-4745-b825-467d8872aaec/volumes" Oct 29 22:49:39.255307 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:49:39.255269 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:49:43.307953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.307922 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk"] Oct 29 22:49:43.308996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.308972 5073 cpu_manager.go:414] "containerMap: removing container" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" Oct 29 22:49:43.308996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.308998 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" Oct 29 22:49:43.309233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.309013 5073 cpu_manager.go:414] "containerMap: removing container" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" Oct 29 22:49:43.309233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.309021 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" Oct 29 22:49:43.309233 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.309228 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" Oct 29 22:49:43.309336 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.309246 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="790bb1ce-002a-4745-b825-467d8872aaec" containerName="istio-proxy" Oct 29 22:49:43.313674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.313649 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.337650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.337621 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk"] Oct 29 22:49:43.344254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.344186 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-e81b56\"/\"kube-root-ca.crt\"" Oct 29 22:49:43.344367 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.344251 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-e81b56\"/\"openshift-service-ca.crt\"" Oct 29 22:49:43.347467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.347448 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-e81b56\"/\"default-dockercfg-bxhg8\"" Oct 29 22:49:43.347572 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.347554 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-e81b56\"/\"sub-test-model-e81b56-kserve-self-signed-certs\"" Oct 29 22:49:43.405029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.405002 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.405183 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.405058 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.405258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.405229 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvg6r\" (UniqueName: \"kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.405445 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.405427 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.405526 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.405487 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.405771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.405719 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.507689 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.507654 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.507733 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.507780 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.507826 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvg6r\" (UniqueName: \"kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.507994 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.508051 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.508264 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.508466 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.508877 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.510999 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.510975 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.511229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.511210 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.529660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.529638 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvg6r\" (UniqueName: \"kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r\") pod \"sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:43.646449 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:43.641432 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:45.104587 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.084137 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk"] Oct 29 22:49:45.609931 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.609895 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:49:45.624856 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.624826 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:49:45.629119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.629098 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:49:45.634992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.634970 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:49:45.673534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.673502 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" event={"ID":"b67510dd-20c1-4c8b-818c-5c7ce6d2022d","Type":"ContainerStarted","Data":"f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221"} Oct 29 22:49:45.673685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.673548 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" event={"ID":"b67510dd-20c1-4c8b-818c-5c7ce6d2022d","Type":"ContainerStarted","Data":"18da9439075ee82e95f235cd6ff5993963fc2f8f03330f2db35c4a3be46c7cb1"} Oct 29 22:49:45.732549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:45.732486 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" podStartSLOduration=2.732469929 podStartE2EDuration="2.732469929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:45.727645493 +0000 UTC m=+2101.043039613" watchObservedRunningTime="2026-10-29 22:49:45.732469929 +0000 UTC m=+2101.047864009" Oct 29 22:49:46.176829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:46.176793 5073 scope.go:122] "RemoveContainer" containerID="2ab366051914f7a57accbb70acea4c8dd568f81c00601c30ac064df709e9e8e1" Oct 29 22:49:46.188901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:46.188879 5073 scope.go:122] "RemoveContainer" containerID="0056318871de6a11c15f8c96ff3376a612cb84ab735e4b4c2ad8441126c427bc" Oct 29 22:49:46.643433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:46.643395 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:53.643400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:53.643366 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:53.657989 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:53.657960 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:49:53.779856 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:49:53.779818 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:50:09.256820 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:50:09.256739 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:50:30.876303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:30.876269 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:50:36.796967 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:36.796932 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9011-dedc99"] Oct 29 22:50:36.802031 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:36.802002 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:36.818391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:36.818359 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9011-dedc99"] Oct 29 22:50:36.826344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:36.826318 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c5k5w\"" Oct 29 22:50:36.902171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:36.902079 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkvw\" (UniqueName: \"kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw\") pod \"test-curl-9011-dedc99\" (UID: \"32406ba8-9778-4a85-ad8f-18e432da8b2a\") " pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:37.003299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:37.003264 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkvw\" (UniqueName: \"kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw\") pod \"test-curl-9011-dedc99\" (UID: \"32406ba8-9778-4a85-ad8f-18e432da8b2a\") " pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:37.025618 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:37.025587 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkvw\" (UniqueName: \"kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw\") pod \"test-curl-9011-dedc99\" (UID: \"32406ba8-9778-4a85-ad8f-18e432da8b2a\") " pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:37.115787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:37.115754 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:37.419627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:37.419598 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9011-dedc99"] Oct 29 22:50:37.420674 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:50:37.420642 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32406ba8_9778_4a85_ad8f_18e432da8b2a.slice/crio-ff333ea6c15f701f4298a79450c46e60c0988b03eed3cee597be079f854db056 WatchSource:0}: Error finding container ff333ea6c15f701f4298a79450c46e60c0988b03eed3cee597be079f854db056: Status 404 returned error can't find the container with id ff333ea6c15f701f4298a79450c46e60c0988b03eed3cee597be079f854db056 Oct 29 22:50:38.220080 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:38.220040 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9011-dedc99" event={"ID":"32406ba8-9778-4a85-ad8f-18e432da8b2a","Type":"ContainerStarted","Data":"0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895"} Oct 29 22:50:38.220604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:38.220093 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9011-dedc99" event={"ID":"32406ba8-9778-4a85-ad8f-18e432da8b2a","Type":"ContainerStarted","Data":"ff333ea6c15f701f4298a79450c46e60c0988b03eed3cee597be079f854db056"} Oct 29 22:50:38.261353 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:38.261299 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9011-dedc99" podStartSLOduration=2.261283776 podStartE2EDuration="2.261283776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:38.259464244 +0000 UTC m=+2153.574858324" watchObservedRunningTime="2026-10-29 22:50:38.261283776 +0000 UTC m=+2153.576677856" Oct 29 22:50:39.148763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:39.148721 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9011-dedc99"] Oct 29 22:50:39.155029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:39.154993 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9011-dedc99"] Oct 29 22:50:39.258570 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:50:39.258534 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:50:40.236835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:40.236791 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9011-dedc99" podUID="32406ba8-9778-4a85-ad8f-18e432da8b2a" containerName="test-curl-9011-dedc99" containerID="cri-o://0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895" gracePeriod=2 Oct 29 22:50:40.252100 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:50:40.252062 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9011-dedc99\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="32406ba8-9778-4a85-ad8f-18e432da8b2a" pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:42.476804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:42.476775 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:42.617722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:42.617629 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw\" (UniqueName: \"kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw\") pod \"32406ba8-9778-4a85-ad8f-18e432da8b2a\" (UID: \"32406ba8-9778-4a85-ad8f-18e432da8b2a\") " Oct 29 22:50:42.620393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:42.620362 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw" pod "32406ba8-9778-4a85-ad8f-18e432da8b2a" (UID: "32406ba8-9778-4a85-ad8f-18e432da8b2a"). InnerVolumeSpecName "kube-api-access-vzkvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:50:42.719745 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:42.719709 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzkvw\" (UniqueName: \"kubernetes.io/projected/32406ba8-9778-4a85-ad8f-18e432da8b2a-kube-api-access-vzkvw\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:50:42.841352 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:42.841320 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32406ba8-9778-4a85-ad8f-18e432da8b2a" path="/var/lib/kubelet/pods/32406ba8-9778-4a85-ad8f-18e432da8b2a/volumes" Oct 29 22:50:43.259630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:43.259594 5073 generic.go:356] "Generic (PLEG): container finished" podID="32406ba8-9778-4a85-ad8f-18e432da8b2a" containerID="0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895" exitCode=137 Oct 29 22:50:43.259838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:43.259639 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-dedc99" Oct 29 22:50:43.259838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:43.259693 5073 scope.go:122] "RemoveContainer" containerID="0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895" Oct 29 22:50:43.270556 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:43.270536 5073 scope.go:122] "RemoveContainer" containerID="0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895" Oct 29 22:50:43.270874 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:50:43.270855 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895\": container with ID starting with 0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895 not found: ID does not exist" containerID="0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895" Oct 29 22:50:43.270936 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:50:43.270885 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895"} err="failed to get container status \"0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895\": rpc error: code = NotFound desc = could not find container \"0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895\": container with ID starting with 0555ced7227e3df31464b8adb51726f69547df0f473d9cc32b8bff7dcba12895 not found: ID does not exist" Oct 29 22:51:02.102469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:02.102427 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8"] Oct 29 22:51:09.257285 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:09.257244 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:51:17.156425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.156378 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" podUID="f93e8566-49c7-4e28-9656-3ec25438df7a" containerName="main" containerID="cri-o://a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725" gracePeriod=45 Oct 29 22:51:17.471369 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.471341 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:51:17.523744 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.523705 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache\") pod \"f93e8566-49c7-4e28-9656-3ec25438df7a\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " Oct 29 22:51:17.523947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.523926 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs\") pod \"f93e8566-49c7-4e28-9656-3ec25438df7a\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " Oct 29 22:51:17.524027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.523973 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm\") pod \"f93e8566-49c7-4e28-9656-3ec25438df7a\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " Oct 29 22:51:17.524027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.524012 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir\") pod \"f93e8566-49c7-4e28-9656-3ec25438df7a\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " Oct 29 22:51:17.524166 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.524118 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp\" (UniqueName: \"kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp\") pod \"f93e8566-49c7-4e28-9656-3ec25438df7a\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " Oct 29 22:51:17.524268 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.524251 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home\") pod \"f93e8566-49c7-4e28-9656-3ec25438df7a\" (UID: \"f93e8566-49c7-4e28-9656-3ec25438df7a\") " Oct 29 22:51:17.525516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.525486 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache" pod "f93e8566-49c7-4e28-9656-3ec25438df7a" (UID: "f93e8566-49c7-4e28-9656-3ec25438df7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:17.525766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.525305 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home" pod "f93e8566-49c7-4e28-9656-3ec25438df7a" (UID: "f93e8566-49c7-4e28-9656-3ec25438df7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:17.526047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.526024 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir" pod "f93e8566-49c7-4e28-9656-3ec25438df7a" (UID: "f93e8566-49c7-4e28-9656-3ec25438df7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:17.529043 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.529002 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp" pod "f93e8566-49c7-4e28-9656-3ec25438df7a" (UID: "f93e8566-49c7-4e28-9656-3ec25438df7a"). InnerVolumeSpecName "kube-api-access-9dlhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:17.529321 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.529288 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm" pod "f93e8566-49c7-4e28-9656-3ec25438df7a" (UID: "f93e8566-49c7-4e28-9656-3ec25438df7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:17.529439 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.529353 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs" pod "f93e8566-49c7-4e28-9656-3ec25438df7a" (UID: "f93e8566-49c7-4e28-9656-3ec25438df7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:17.627128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.627090 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93e8566-49c7-4e28-9656-3ec25438df7a-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.627128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.627122 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.627128 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.627131 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.627432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.627139 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dlhp\" (UniqueName: \"kubernetes.io/projected/f93e8566-49c7-4e28-9656-3ec25438df7a-kube-api-access-9dlhp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.627432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.627161 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.627432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.627170 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93e8566-49c7-4e28-9656-3ec25438df7a-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.707029 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.706998 5073 generic.go:356] "Generic (PLEG): container finished" podID="f93e8566-49c7-4e28-9656-3ec25438df7a" containerID="a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725" exitCode=0 Oct 29 22:51:17.707229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.707064 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" Oct 29 22:51:17.707229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.707083 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" event={"ID":"f93e8566-49c7-4e28-9656-3ec25438df7a","Type":"ContainerDied","Data":"a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725"} Oct 29 22:51:17.707229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.707133 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8" event={"ID":"f93e8566-49c7-4e28-9656-3ec25438df7a","Type":"ContainerDied","Data":"96b20e8786d5871ca7a72c8cd7d4ad2e78c3805bfdb339b2a8cbd9b8f292c4de"} Oct 29 22:51:17.707229 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.707189 5073 scope.go:122] "RemoveContainer" containerID="a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725" Oct 29 22:51:17.720727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.720702 5073 scope.go:122] "RemoveContainer" containerID="a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725" Oct 29 22:51:17.721084 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:17.721060 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725\": container with ID starting with a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725 not found: ID does not exist" containerID="a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725" Oct 29 22:51:17.721168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.721095 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725"} err="failed to get container status \"a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725\": rpc error: code = NotFound desc = could not find container \"a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725\": container with ID starting with a04cdb78e755f66d1641c9bbcdfcab71b059499bb91eb9fbb3870c210fce6725 not found: ID does not exist" Oct 29 22:51:17.764489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.764454 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8"] Oct 29 22:51:17.768168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:17.768119 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/auth-test-model-afdbe8-kserve-7cfd89bd9c-786t8"] Oct 29 22:51:18.840436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:18.840403 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93e8566-49c7-4e28-9656-3ec25438df7a" path="/var/lib/kubelet/pods/f93e8566-49c7-4e28-9656-3ec25438df7a/volumes" Oct 29 22:51:21.768964 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:21.768920 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t"] Oct 29 22:51:21.769477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:21.769175 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" containerID="cri-o://3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c" gracePeriod=30 Oct 29 22:51:21.949671 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:21.949635 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7"] Oct 29 22:51:21.949887 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:21.949854 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" containerID="cri-o://d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee" gracePeriod=30 Oct 29 22:51:22.128924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:22.128779 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45"] Oct 29 22:51:22.129118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:22.129089 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" containerID="cri-o://1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3" gracePeriod=30 Oct 29 22:51:25.564218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:25.563976 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22"] Oct 29 22:51:26.810583 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:26.810554 5073 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:51:26.811011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:26.810620 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:51:27.002359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.002326 5073 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:51:27.002548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.002384 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:51:27.005968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.005845 5073 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:51:27.005968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.005916 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:51:27.087378 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.087349 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:51:27.214274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214238 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214481 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214352 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214481 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214455 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214515 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214548 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214578 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.214611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214595 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214680 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214812 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214843 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy\") pod \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\" (UID: \"2c4a3856-30ab-4409-8c5b-dc32b1c40d84\") " Oct 29 22:51:27.214941 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.214921 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:27.215632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.215280 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.215768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.215677 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.215942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.215910 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.216511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.216420 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.216511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.216447 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.216511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.216462 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.216511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.216476 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.216511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.216489 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.217987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.217945 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.219263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.219233 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:27.219498 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.219469 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "kube-api-access-5tl2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.220590 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.220569 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token" pod "2c4a3856-30ab-4409-8c5b-dc32b1c40d84" (UID: "2c4a3856-30ab-4409-8c5b-dc32b1c40d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.247035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.246997 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:51:27.323898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.320920 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.323898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.320958 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tl2x\" (UniqueName: \"kubernetes.io/projected/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-kube-api-access-5tl2x\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.323898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.320974 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.323898 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.320996 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4a3856-30ab-4409-8c5b-dc32b1c40d84-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.422490 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422455 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.422668 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422528 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.422668 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422563 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.422755 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422734 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.422828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422796 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.422944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.422944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422902 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.422944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422927 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.423165 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.422980 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.423165 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.423051 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.423258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.423142 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:27.423258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.423171 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data\") pod \"8c80d264-862f-4f44-8732-576fc9d67a22\" (UID: \"8c80d264-862f-4f44-8732-576fc9d67a22\") " Oct 29 22:51:27.423353 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.423324 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.423549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.423367 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.424144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.424122 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.424289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.424200 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.424289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.424217 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c80d264-862f-4f44-8732-576fc9d67a22-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.424289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.424231 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.424289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.424245 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.425740 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.425545 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.426107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.426082 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:27.426216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.426130 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.426533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.426518 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j" pod "8c80d264-862f-4f44-8732-576fc9d67a22" (UID: "8c80d264-862f-4f44-8732-576fc9d67a22"). InnerVolumeSpecName "kube-api-access-ztg9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.450244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.450213 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:51:27.525734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.525696 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c80d264-862f-4f44-8732-576fc9d67a22-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.525734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.525731 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.525734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.525739 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c80d264-862f-4f44-8732-576fc9d67a22-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.525980 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.525747 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztg9j\" (UniqueName: \"kubernetes.io/projected/8c80d264-862f-4f44-8732-576fc9d67a22-kube-api-access-ztg9j\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.627016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.626922 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627064 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627178 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627264 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627342 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627376 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627469 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627443 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627508 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627549 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs\") pod \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\" (UID: \"7dd77fe7-3293-4b49-9ca6-53e17e5f56be\") " Oct 29 22:51:27.627650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627597 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.627760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627730 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.627918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627898 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.628003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.627912 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.628282 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.628081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:27.628520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.628495 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.628591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.628526 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.628591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.628541 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.628591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.628556 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.628591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.628568 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.629621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.629598 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:27.629750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.629735 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.629905 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.629882 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "kube-api-access-shw7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.631059 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.631036 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy" pod "7dd77fe7-3293-4b49-9ca6-53e17e5f56be" (UID: "7dd77fe7-3293-4b49-9ca6-53e17e5f56be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.730531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.730496 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.730531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.730528 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.730531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.730538 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shw7m\" (UniqueName: \"kubernetes.io/projected/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-kube-api-access-shw7m\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.730782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.730547 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd77fe7-3293-4b49-9ca6-53e17e5f56be-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.827047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.827013 5073 generic.go:356] "Generic (PLEG): container finished" podID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerID="3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c" exitCode=0 Oct 29 22:51:27.827514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.827100 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" Oct 29 22:51:27.827514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.827105 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" event={"ID":"2c4a3856-30ab-4409-8c5b-dc32b1c40d84","Type":"ContainerDied","Data":"3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c"} Oct 29 22:51:27.827514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.827223 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t" event={"ID":"2c4a3856-30ab-4409-8c5b-dc32b1c40d84","Type":"ContainerDied","Data":"7893bef968a282af3dade9e321d95d197e374b6674a172152af750076ba7e4b4"} Oct 29 22:51:27.827514 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.827257 5073 scope.go:122] "RemoveContainer" containerID="3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c" Oct 29 22:51:27.828763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.828730 5073 generic.go:356] "Generic (PLEG): container finished" podID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerID="1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3" exitCode=0 Oct 29 22:51:27.828882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.828824 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" event={"ID":"7dd77fe7-3293-4b49-9ca6-53e17e5f56be","Type":"ContainerDied","Data":"1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3"} Oct 29 22:51:27.828882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.828874 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" event={"ID":"7dd77fe7-3293-4b49-9ca6-53e17e5f56be","Type":"ContainerDied","Data":"9e6d0d16a0099842be2303ffabde62352823ccf206b9f8926c9f77d6e4b0851c"} Oct 29 22:51:27.829012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.828948 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45" Oct 29 22:51:27.830501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.830477 5073 generic.go:356] "Generic (PLEG): container finished" podID="8c80d264-862f-4f44-8732-576fc9d67a22" containerID="d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee" exitCode=0 Oct 29 22:51:27.831121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.830672 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" event={"ID":"8c80d264-862f-4f44-8732-576fc9d67a22","Type":"ContainerDied","Data":"d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee"} Oct 29 22:51:27.831121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.830711 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" event={"ID":"8c80d264-862f-4f44-8732-576fc9d67a22","Type":"ContainerDied","Data":"366723c66ccb2da11e4a9594376744737763b1c661042de7b8b987b55eb55969"} Oct 29 22:51:27.831121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.830797 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7" Oct 29 22:51:27.839554 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.839524 5073 scope.go:122] "RemoveContainer" containerID="3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c" Oct 29 22:51:27.839930 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:27.839907 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c\": container with ID starting with 3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c not found: ID does not exist" containerID="3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c" Oct 29 22:51:27.840022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.839938 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c"} err="failed to get container status \"3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c\": rpc error: code = NotFound desc = could not find container \"3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c\": container with ID starting with 3f32e0e5b4b3db524a3b9def755c5e5bb515a157b9d76e60b552f4c3b1929d6c not found: ID does not exist" Oct 29 22:51:27.840022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.839956 5073 scope.go:122] "RemoveContainer" containerID="1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3" Oct 29 22:51:27.850901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.850876 5073 scope.go:122] "RemoveContainer" containerID="1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3" Oct 29 22:51:27.851228 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:27.851202 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3\": container with ID starting with 1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3 not found: ID does not exist" containerID="1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3" Oct 29 22:51:27.851309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.851234 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3"} err="failed to get container status \"1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3\": rpc error: code = NotFound desc = could not find container \"1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3\": container with ID starting with 1874d801f60979b67af5fff9a69aa7f1f0ecce4288b72dd6e0f7f13bef0be0b3 not found: ID does not exist" Oct 29 22:51:27.851309 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.851253 5073 scope.go:122] "RemoveContainer" containerID="d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee" Oct 29 22:51:27.861092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.861073 5073 scope.go:122] "RemoveContainer" containerID="d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee" Oct 29 22:51:27.861475 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:27.861450 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee\": container with ID starting with d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee not found: ID does not exist" containerID="d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee" Oct 29 22:51:27.861553 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.861482 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee"} err="failed to get container status \"d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee\": rpc error: code = NotFound desc = could not find container \"d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee\": container with ID starting with d97a55bd25eaf6e1960569266440b49ce9dae64eff4985089a2f1c95da8563ee not found: ID does not exist" Oct 29 22:51:27.885000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.884920 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7"] Oct 29 22:51:27.891127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.891101 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-5cqz7"] Oct 29 22:51:27.927316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.927284 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45"] Oct 29 22:51:27.932557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.932528 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-jjc45"] Oct 29 22:51:27.982297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.980998 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t"] Oct 29 22:51:27.986774 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:27.986680 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-gxp2t"] Oct 29 22:51:28.843633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:28.843603 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" path="/var/lib/kubelet/pods/2c4a3856-30ab-4409-8c5b-dc32b1c40d84/volumes" Oct 29 22:51:28.844206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:28.844189 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" path="/var/lib/kubelet/pods/7dd77fe7-3293-4b49-9ca6-53e17e5f56be/volumes" Oct 29 22:51:28.844672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:28.844657 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" path="/var/lib/kubelet/pods/8c80d264-862f-4f44-8732-576fc9d67a22/volumes" Oct 29 22:51:30.532218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532172 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg"] Oct 29 22:51:30.532830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532815 5073 cpu_manager.go:414] "containerMap: removing container" podUID="32406ba8-9778-4a85-ad8f-18e432da8b2a" containerName="test-curl-9011-dedc99" Oct 29 22:51:30.532879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532830 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32406ba8-9778-4a85-ad8f-18e432da8b2a" containerName="test-curl-9011-dedc99" Oct 29 22:51:30.532879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532845 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" Oct 29 22:51:30.532879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532849 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" Oct 29 22:51:30.532879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532860 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f93e8566-49c7-4e28-9656-3ec25438df7a" containerName="main" Oct 29 22:51:30.532879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532864 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93e8566-49c7-4e28-9656-3ec25438df7a" containerName="main" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532889 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532893 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532909 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.532913 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.533002 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93e8566-49c7-4e28-9656-3ec25438df7a" containerName="main" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.533012 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4a3856-30ab-4409-8c5b-dc32b1c40d84" containerName="istio-proxy" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.533022 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd77fe7-3293-4b49-9ca6-53e17e5f56be" containerName="istio-proxy" Oct 29 22:51:30.533034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.533032 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="32406ba8-9778-4a85-ad8f-18e432da8b2a" containerName="test-curl-9011-dedc99" Oct 29 22:51:30.533295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.533038 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c80d264-862f-4f44-8732-576fc9d67a22" containerName="istio-proxy" Oct 29 22:51:30.538375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.538351 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.558281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.558251 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8ebc60-openshift-default-dockercfg-tvk92\"" Oct 29 22:51:30.576097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.576055 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg"] Oct 29 22:51:30.605534 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.605498 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.605741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.605550 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.605741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.605660 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.605886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.605762 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.605886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.605799 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.605886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.605830 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.606106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.606084 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.606140 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.606125 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mp68\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.606217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.606182 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.707565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707529 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.707565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707580 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.707766 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707725 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.707833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707790 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.707833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707815 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.707930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707839 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.707981 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708046 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708062 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708092 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mp68\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708126 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708185 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708268 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.708782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.708756 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.710575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.710553 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.710707 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.710687 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.728615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.728588 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.728792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.728662 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mp68\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68\") pod \"e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:30.856434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:30.856347 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:31.186066 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.186021 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg"] Oct 29 22:51:31.186238 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:51:31.186213 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad4974f_42cf_49da_9f4a_a39415cfcb3f.slice/crio-5a24de6af9114b06de4fa3c6f62b1267116fcf6c70dd2ffbc0e0241632844541 WatchSource:0}: Error finding container 5a24de6af9114b06de4fa3c6f62b1267116fcf6c70dd2ffbc0e0241632844541: Status 404 returned error can't find the container with id 5a24de6af9114b06de4fa3c6f62b1267116fcf6c70dd2ffbc0e0241632844541 Oct 29 22:51:31.189144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.189102 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:51:31.189279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.189206 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:51:31.189279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.189256 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:51:31.876561 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.876518 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" event={"ID":"3ad4974f-42cf-49da-9f4a-a39415cfcb3f","Type":"ContainerStarted","Data":"1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7"} Oct 29 22:51:31.876979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.876591 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" event={"ID":"3ad4974f-42cf-49da-9f4a-a39415cfcb3f","Type":"ContainerStarted","Data":"5a24de6af9114b06de4fa3c6f62b1267116fcf6c70dd2ffbc0e0241632844541"} Oct 29 22:51:31.956871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:31.956810 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podStartSLOduration=1.9567952709999998 podStartE2EDuration="1.956795271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:31.93965651 +0000 UTC m=+2207.255050589" watchObservedRunningTime="2026-10-29 22:51:31.956795271 +0000 UTC m=+2207.272189415" Oct 29 22:51:32.856976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:32.856943 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:32.858559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:32.858535 5073 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 22:51:32.858682 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:32.858586 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 22:51:33.856913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:33.856878 5073 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 22:51:33.857426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:33.856951 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 22:51:34.857169 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:34.857116 5073 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 22:51:34.857680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:34.857197 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 22:51:35.857642 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:35.857605 5073 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 22:51:35.858087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:35.857664 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 22:51:36.857464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:36.857378 5073 patch_prober.go:29] interesting pod/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 22:51:36.857464 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:36.857437 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 22:51:37.341206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:37.341163 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq"] Oct 29 22:51:37.341653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:37.341396 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" containerID="cri-o://47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8" gracePeriod=30 Oct 29 22:51:37.930560 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:37.930528 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:37.931451 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:37.931426 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:37.932873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:37.932848 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:51:39.257982 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:39.257944 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:51:40.594474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.594398 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" podUID="5391ed24-156b-4131-9964-1c8ab042cc2c" containerName="main" containerID="cri-o://5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454" gracePeriod=45 Oct 29 22:51:40.906521 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.906492 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:51:40.937549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.937512 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache\") pod \"5391ed24-156b-4131-9964-1c8ab042cc2c\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " Oct 29 22:51:40.937787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.937645 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7\" (UniqueName: \"kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7\") pod \"5391ed24-156b-4131-9964-1c8ab042cc2c\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " Oct 29 22:51:40.937787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.937671 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home\") pod \"5391ed24-156b-4131-9964-1c8ab042cc2c\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " Oct 29 22:51:40.937787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.937766 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs\") pod \"5391ed24-156b-4131-9964-1c8ab042cc2c\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " Oct 29 22:51:40.937995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.937782 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache" pod "5391ed24-156b-4131-9964-1c8ab042cc2c" (UID: "5391ed24-156b-4131-9964-1c8ab042cc2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:40.937995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.937966 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir\") pod \"5391ed24-156b-4131-9964-1c8ab042cc2c\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " Oct 29 22:51:40.938109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.938000 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm\") pod \"5391ed24-156b-4131-9964-1c8ab042cc2c\" (UID: \"5391ed24-156b-4131-9964-1c8ab042cc2c\") " Oct 29 22:51:40.938109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.938061 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home" pod "5391ed24-156b-4131-9964-1c8ab042cc2c" (UID: "5391ed24-156b-4131-9964-1c8ab042cc2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:40.938700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.938675 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir" pod "5391ed24-156b-4131-9964-1c8ab042cc2c" (UID: "5391ed24-156b-4131-9964-1c8ab042cc2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:40.939090 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.939065 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.939274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.939094 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.939274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.939110 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:40.940667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.940639 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7" pod "5391ed24-156b-4131-9964-1c8ab042cc2c" (UID: "5391ed24-156b-4131-9964-1c8ab042cc2c"). InnerVolumeSpecName "kube-api-access-rhvx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:40.940871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.940848 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs" pod "5391ed24-156b-4131-9964-1c8ab042cc2c" (UID: "5391ed24-156b-4131-9964-1c8ab042cc2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:40.940927 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.940894 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm" pod "5391ed24-156b-4131-9964-1c8ab042cc2c" (UID: "5391ed24-156b-4131-9964-1c8ab042cc2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:40.967079 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.967028 5073 generic.go:356] "Generic (PLEG): container finished" podID="5391ed24-156b-4131-9964-1c8ab042cc2c" containerID="5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454" exitCode=0 Oct 29 22:51:40.967287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.967161 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" Oct 29 22:51:40.967287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.967227 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" event={"ID":"5391ed24-156b-4131-9964-1c8ab042cc2c","Type":"ContainerDied","Data":"5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454"} Oct 29 22:51:40.967287 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.967279 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22" event={"ID":"5391ed24-156b-4131-9964-1c8ab042cc2c","Type":"ContainerDied","Data":"bacb004449f4a336521065edd6046772060ae4ff0dd83f24ea7227b1c4d4e787"} Oct 29 22:51:40.967443 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.967305 5073 scope.go:122] "RemoveContainer" containerID="5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454" Oct 29 22:51:40.991631 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.991600 5073 scope.go:122] "RemoveContainer" containerID="5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454" Oct 29 22:51:40.992084 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:40.992046 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454\": container with ID starting with 5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454 not found: ID does not exist" containerID="5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454" Oct 29 22:51:40.992215 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:40.992097 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454"} err="failed to get container status \"5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454\": rpc error: code = NotFound desc = could not find container \"5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454\": container with ID starting with 5793e1713d221223a6d8f3865b7fdf15e1492c7705a6061e29885d35445ad454 not found: ID does not exist" Oct 29 22:51:41.026695 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:41.026660 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22"] Oct 29 22:51:41.030286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:41.030258 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/auth-test-model-84304f-kserve-84f5444469-tvp22"] Oct 29 22:51:41.040910 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:41.040881 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5391ed24-156b-4131-9964-1c8ab042cc2c-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:41.040910 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:41.040911 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvx7\" (UniqueName: \"kubernetes.io/projected/5391ed24-156b-4131-9964-1c8ab042cc2c-kube-api-access-rhvx7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:41.041096 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:41.040924 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391ed24-156b-4131-9964-1c8ab042cc2c-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.198603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.198565 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:51:42.199007 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.198640 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:51:42.639763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.639727 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:51:42.692592 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692554 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.692792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692698 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.692792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692730 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.692918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692792 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.692918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692828 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:42.692918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692857 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.692918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.692883 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.693193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693028 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.693193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693036 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:42.693193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693066 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.693193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693121 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb\") pod \"7559b915-afb7-4ceb-974b-8122f6f77760\" (UID: \"7559b915-afb7-4ceb-974b-8122f6f77760\") " Oct 29 22:51:42.693420 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693216 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:42.693481 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693446 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:42.693559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.693536 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:42.694428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.694402 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.694562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.694431 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7559b915-afb7-4ceb-974b-8122f6f77760-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.694562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.694445 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.694562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.694458 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.694562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.694470 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.695866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.695829 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:42.695985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.695915 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:42.696143 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.696120 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "kube-api-access-hxrfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:42.696288 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.696264 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token" pod "7559b915-afb7-4ceb-974b-8122f6f77760" (UID: "7559b915-afb7-4ceb-974b-8122f6f77760"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:42.796472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.796378 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.796472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.796417 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrfb\" (UniqueName: \"kubernetes.io/projected/7559b915-afb7-4ceb-974b-8122f6f77760-kube-api-access-hxrfb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.796472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.796437 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7559b915-afb7-4ceb-974b-8122f6f77760-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.796472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.796452 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7559b915-afb7-4ceb-974b-8122f6f77760-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:42.839947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.839909 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5391ed24-156b-4131-9964-1c8ab042cc2c" path="/var/lib/kubelet/pods/5391ed24-156b-4131-9964-1c8ab042cc2c/volumes" Oct 29 22:51:42.983530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.983498 5073 generic.go:356] "Generic (PLEG): container finished" podID="7559b915-afb7-4ceb-974b-8122f6f77760" containerID="47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8" exitCode=0 Oct 29 22:51:42.983735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.983548 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" Oct 29 22:51:42.983735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.983582 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" event={"ID":"7559b915-afb7-4ceb-974b-8122f6f77760","Type":"ContainerDied","Data":"47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8"} Oct 29 22:51:42.983735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.983646 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq" event={"ID":"7559b915-afb7-4ceb-974b-8122f6f77760","Type":"ContainerDied","Data":"6610e1f0f5f3b145507947eeab164e560de8e4cd2b334cf6cdcc38a097f682a2"} Oct 29 22:51:42.983735 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.983676 5073 scope.go:122] "RemoveContainer" containerID="47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8" Oct 29 22:51:42.994876 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.994851 5073 scope.go:122] "RemoveContainer" containerID="47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8" Oct 29 22:51:42.995247 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:42.995216 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8\": container with ID starting with 47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8 not found: ID does not exist" containerID="47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8" Oct 29 22:51:42.995336 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:42.995261 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8"} err="failed to get container status \"47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8\": rpc error: code = NotFound desc = could not find container \"47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8\": container with ID starting with 47d45978535fe7f40cd1b42a938676f6f441734cd689bddd441034780aa2ade8 not found: ID does not exist" Oct 29 22:51:43.044436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:43.044402 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq"] Oct 29 22:51:43.058109 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:43.058023 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-6b5qq"] Oct 29 22:51:44.123643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.123603 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn"] Oct 29 22:51:44.124068 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.123898 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" podUID="3546bd90-f494-41cb-97a5-f53282656cd8" containerName="discovery" containerID="cri-o://12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f" gracePeriod=30 Oct 29 22:51:44.436699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.436672 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:51:44.522277 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522236 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.522491 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522323 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.522491 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522398 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.522608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522494 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.522608 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522544 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.522714 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522629 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:44.522770 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522712 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.522828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.522781 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl\") pod \"3546bd90-f494-41cb-97a5-f53282656cd8\" (UID: \"3546bd90-f494-41cb-97a5-f53282656cd8\") " Oct 29 22:51:44.523661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.523625 5073 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-ca-configmap\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.525511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.525484 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:44.525511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.525502 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:44.525696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.525576 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:44.525696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.525639 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:44.525696 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.525682 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:44.525877 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.525855 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl" pod "3546bd90-f494-41cb-97a5-f53282656cd8" (UID: "3546bd90-f494-41cb-97a5-f53282656cd8"). InnerVolumeSpecName "kube-api-access-6wcwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:44.625575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.625534 5073 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-cacerts\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.625575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.625567 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.625575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.625578 5073 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-kubeconfig\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.625575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.625586 5073 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3546bd90-f494-41cb-97a5-f53282656cd8-local-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.625953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.625595 5073 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3546bd90-f494-41cb-97a5-f53282656cd8-istio-csr-dns-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.625953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.625603 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcwl\" (UniqueName: \"kubernetes.io/projected/3546bd90-f494-41cb-97a5-f53282656cd8-kube-api-access-6wcwl\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:51:44.867074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:44.863767 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" path="/var/lib/kubelet/pods/7559b915-afb7-4ceb-974b-8122f6f77760/volumes" Oct 29 22:51:45.001603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.001565 5073 generic.go:356] "Generic (PLEG): container finished" podID="3546bd90-f494-41cb-97a5-f53282656cd8" containerID="12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f" exitCode=0 Oct 29 22:51:45.001831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.001622 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" Oct 29 22:51:45.001831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.001647 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" event={"ID":"3546bd90-f494-41cb-97a5-f53282656cd8","Type":"ContainerDied","Data":"12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f"} Oct 29 22:51:45.001831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.001704 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn" event={"ID":"3546bd90-f494-41cb-97a5-f53282656cd8","Type":"ContainerDied","Data":"adf2f99136cc86edeecaba6c49d8e69ae153603c220b78a56d2fc123f50fa9d7"} Oct 29 22:51:45.001831 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.001734 5073 scope.go:122] "RemoveContainer" containerID="12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f" Oct 29 22:51:45.014394 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.014358 5073 scope.go:122] "RemoveContainer" containerID="12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f" Oct 29 22:51:45.014720 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:51:45.014699 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f\": container with ID starting with 12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f not found: ID does not exist" containerID="12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f" Oct 29 22:51:45.014793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.014730 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f"} err="failed to get container status \"12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f\": rpc error: code = NotFound desc = could not find container \"12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f\": container with ID starting with 12ef3a4e86a6492786eee1bfb12f9c30fa03206993ac987692318dbf9e22f15f not found: ID does not exist" Oct 29 22:51:45.061325 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.061290 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn"] Oct 29 22:51:45.074908 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:45.074880 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-st7pn"] Oct 29 22:51:46.840648 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:46.840615 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3546bd90-f494-41cb-97a5-f53282656cd8" path="/var/lib/kubelet/pods/3546bd90-f494-41cb-97a5-f53282656cd8/volumes" Oct 29 22:51:52.252249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:51:52.252212 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz"] Oct 29 22:52:00.032907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.032870 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z"] Oct 29 22:52:00.033569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033551 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" Oct 29 22:52:00.033569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033569 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" Oct 29 22:52:00.033697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033598 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3546bd90-f494-41cb-97a5-f53282656cd8" containerName="discovery" Oct 29 22:52:00.033697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033603 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3546bd90-f494-41cb-97a5-f53282656cd8" containerName="discovery" Oct 29 22:52:00.033697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033621 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5391ed24-156b-4131-9964-1c8ab042cc2c" containerName="main" Oct 29 22:52:00.033697 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033626 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5391ed24-156b-4131-9964-1c8ab042cc2c" containerName="main" Oct 29 22:52:00.033923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033717 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3546bd90-f494-41cb-97a5-f53282656cd8" containerName="discovery" Oct 29 22:52:00.033923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033730 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5391ed24-156b-4131-9964-1c8ab042cc2c" containerName="main" Oct 29 22:52:00.033923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.033743 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7559b915-afb7-4ceb-974b-8122f6f77760" containerName="istio-proxy" Oct 29 22:52:00.039177 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.039133 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.061213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.060216 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z"] Oct 29 22:52:00.068902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.068873 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"rate-test-model-afdbe8-kserve-self-signed-certs\"" Oct 29 22:52:00.069103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.068964 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"default-dockercfg-qfqv6\"" Oct 29 22:52:00.069218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.069191 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"openshift-service-ca.crt\"" Oct 29 22:52:00.069304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.069222 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-afdbe8\"/\"kube-root-ca.crt\"" Oct 29 22:52:00.095646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.095611 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.095836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.095762 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.095836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.095829 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4php5\" (UniqueName: \"kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.095957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.095941 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.096087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.096060 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.096263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.096104 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198430 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198543 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4php5\" (UniqueName: \"kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198655 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198774 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198815 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198915 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198865 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.198969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.198944 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.199226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.199139 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.199375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.199227 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.201253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.201224 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.201850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.201827 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.220844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.220811 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4php5\" (UniqueName: \"kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5\") pod \"rate-test-model-afdbe8-kserve-6b54547b9-2sr9z\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.360659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.360561 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:00.718537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:00.718501 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z"] Oct 29 22:52:00.719258 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:52:00.719230 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d776659_9fd8_4dde_854d_5743feca1069.slice/crio-57ba8350a5b71fa029248fa4cec084a9004770514cb10fa596dbf7215d2a24a9 WatchSource:0}: Error finding container 57ba8350a5b71fa029248fa4cec084a9004770514cb10fa596dbf7215d2a24a9: Status 404 returned error can't find the container with id 57ba8350a5b71fa029248fa4cec084a9004770514cb10fa596dbf7215d2a24a9 Oct 29 22:52:01.141119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:01.141018 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" event={"ID":"7d776659-9fd8-4dde-854d-5743feca1069","Type":"ContainerStarted","Data":"ca5ac928ff0bbfb25de7ff924a8148f7a0faff44c444da59a963f33561e4e1f4"} Oct 29 22:52:01.141119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:01.141068 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" event={"ID":"7d776659-9fd8-4dde-854d-5743feca1069","Type":"ContainerStarted","Data":"57ba8350a5b71fa029248fa4cec084a9004770514cb10fa596dbf7215d2a24a9"} Oct 29 22:52:01.188431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:01.188364 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" podStartSLOduration=1.188341644 podStartE2EDuration="1.188341644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:01.185491345 +0000 UTC m=+2236.500885424" watchObservedRunningTime="2026-10-29 22:52:01.188341644 +0000 UTC m=+2236.503735726" Oct 29 22:52:01.361682 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:01.361641 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:04.465450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:04.465418 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:52:07.290859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.290805 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" podUID="097780fe-e0f8-408a-bd92-b8bf41d881ad" containerName="main" containerID="cri-o://95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb" gracePeriod=45 Oct 29 22:52:07.547660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.547566 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" podUID="097780fe-e0f8-408a-bd92-b8bf41d881ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": dial tcp 10.132.0.145:8000: connect: connection refused" Oct 29 22:52:07.721441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.721406 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:52:07.867001 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.866914 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home\") pod \"097780fe-e0f8-408a-bd92-b8bf41d881ad\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " Oct 29 22:52:07.867217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867038 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm\") pod \"097780fe-e0f8-408a-bd92-b8bf41d881ad\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " Oct 29 22:52:07.867217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867132 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl\" (UniqueName: \"kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl\") pod \"097780fe-e0f8-408a-bd92-b8bf41d881ad\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " Oct 29 22:52:07.867217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867181 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir\") pod \"097780fe-e0f8-408a-bd92-b8bf41d881ad\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " Oct 29 22:52:07.867425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867252 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home" pod "097780fe-e0f8-408a-bd92-b8bf41d881ad" (UID: "097780fe-e0f8-408a-bd92-b8bf41d881ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:07.867425 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867292 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs\") pod \"097780fe-e0f8-408a-bd92-b8bf41d881ad\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " Oct 29 22:52:07.867540 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867476 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache\") pod \"097780fe-e0f8-408a-bd92-b8bf41d881ad\" (UID: \"097780fe-e0f8-408a-bd92-b8bf41d881ad\") " Oct 29 22:52:07.867601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867540 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir" pod "097780fe-e0f8-408a-bd92-b8bf41d881ad" (UID: "097780fe-e0f8-408a-bd92-b8bf41d881ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:07.867872 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.867843 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache" pod "097780fe-e0f8-408a-bd92-b8bf41d881ad" (UID: "097780fe-e0f8-408a-bd92-b8bf41d881ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:07.868198 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.868176 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.868293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.868225 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.868293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.868241 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.870253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.870102 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs" pod "097780fe-e0f8-408a-bd92-b8bf41d881ad" (UID: "097780fe-e0f8-408a-bd92-b8bf41d881ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:07.870253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.870219 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm" pod "097780fe-e0f8-408a-bd92-b8bf41d881ad" (UID: "097780fe-e0f8-408a-bd92-b8bf41d881ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:07.870415 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.870386 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl" pod "097780fe-e0f8-408a-bd92-b8bf41d881ad" (UID: "097780fe-e0f8-408a-bd92-b8bf41d881ad"). InnerVolumeSpecName "kube-api-access-b7gkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:07.982925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.982880 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097780fe-e0f8-408a-bd92-b8bf41d881ad-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.983144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.982999 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7gkl\" (UniqueName: \"kubernetes.io/projected/097780fe-e0f8-408a-bd92-b8bf41d881ad-kube-api-access-b7gkl\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.983144 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:07.983052 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097780fe-e0f8-408a-bd92-b8bf41d881ad-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:08.239056 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.239022 5073 generic.go:356] "Generic (PLEG): container finished" podID="097780fe-e0f8-408a-bd92-b8bf41d881ad" containerID="95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb" exitCode=0 Oct 29 22:52:08.239308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.239107 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" Oct 29 22:52:08.239308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.239109 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" event={"ID":"097780fe-e0f8-408a-bd92-b8bf41d881ad","Type":"ContainerDied","Data":"95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb"} Oct 29 22:52:08.239308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.239249 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz" event={"ID":"097780fe-e0f8-408a-bd92-b8bf41d881ad","Type":"ContainerDied","Data":"60f193628c61f43fcd4091f749b92d606cc3aa56e20df6f1dbaae430257fec3c"} Oct 29 22:52:08.239308 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.239280 5073 scope.go:122] "RemoveContainer" containerID="95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb" Oct 29 22:52:08.251570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.251531 5073 scope.go:122] "RemoveContainer" containerID="95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb" Oct 29 22:52:08.251889 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:52:08.251861 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb\": container with ID starting with 95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb not found: ID does not exist" containerID="95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb" Oct 29 22:52:08.251972 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.251899 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb"} err="failed to get container status \"95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb\": rpc error: code = NotFound desc = could not find container \"95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb\": container with ID starting with 95abd6a527b250006baf63255b8d2670a9c407ff790a71113f6e7ed6e06418eb not found: ID does not exist" Oct 29 22:52:08.297852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.297819 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz"] Oct 29 22:52:08.310239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.310208 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-172c10/test-model-172c10-kserve-68f84c94c9-w27sz"] Oct 29 22:52:08.840971 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:08.840923 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097780fe-e0f8-408a-bd92-b8bf41d881ad" path="/var/lib/kubelet/pods/097780fe-e0f8-408a-bd92-b8bf41d881ad/volumes" Oct 29 22:52:09.261719 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:52:09.261674 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:52:10.361844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:10.361810 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:10.375314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:10.375281 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:11.293412 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:11.293381 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:52:24.398087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.398042 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx"] Oct 29 22:52:24.398866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.398844 5073 cpu_manager.go:414] "containerMap: removing container" podUID="097780fe-e0f8-408a-bd92-b8bf41d881ad" containerName="main" Oct 29 22:52:24.398914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.398868 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097780fe-e0f8-408a-bd92-b8bf41d881ad" containerName="main" Oct 29 22:52:24.398969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.398960 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="097780fe-e0f8-408a-bd92-b8bf41d881ad" containerName="main" Oct 29 22:52:24.403914 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.403891 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.423873 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.423841 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx"] Oct 29 22:52:24.433392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.433359 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-437981-openshift-default-dockercfg-d74jz\"" Oct 29 22:52:24.502351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502310 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502374 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502486 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502549 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502611 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502588 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfdq\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502634 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502715 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502698 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502804 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502790 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.502840 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.502821 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.603807 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.603770 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.603995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.603852 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.603995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.603880 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.603995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.603942 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.603995 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.603984 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604046 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604111 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604135 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfdq\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604380 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604344 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604413 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604470 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604554 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604536 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604844 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.604994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.604894 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.606895 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.606872 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.606973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.606916 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.627391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.627358 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.628429 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.628403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfdq\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:24.718833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:24.718802 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:25.046639 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.046600 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx"] Oct 29 22:52:25.047270 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:52:25.047244 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256cffb1_0fc4_42f0_8496_4cd71c3a84c9.slice/crio-bbf05a9db7935895b1e1a116d017b408210dce4fc639de8517c730f4995d7a6e WatchSource:0}: Error finding container bbf05a9db7935895b1e1a116d017b408210dce4fc639de8517c730f4995d7a6e: Status 404 returned error can't find the container with id bbf05a9db7935895b1e1a116d017b408210dce4fc639de8517c730f4995d7a6e Oct 29 22:52:25.050313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.050277 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:52:25.050410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.050360 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:52:25.050410 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.050405 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:52:25.391647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.391539 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" event={"ID":"256cffb1-0fc4-42f0-8496-4cd71c3a84c9","Type":"ContainerStarted","Data":"a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605"} Oct 29 22:52:25.391647 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.391609 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" event={"ID":"256cffb1-0fc4-42f0-8496-4cd71c3a84c9","Type":"ContainerStarted","Data":"bbf05a9db7935895b1e1a116d017b408210dce4fc639de8517c730f4995d7a6e"} Oct 29 22:52:25.440010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.439945 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podStartSLOduration=1.439926018 podStartE2EDuration="1.439926018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:25.436445563 +0000 UTC m=+2260.751839656" watchObservedRunningTime="2026-10-29 22:52:25.439926018 +0000 UTC m=+2260.755320099" Oct 29 22:52:25.719132 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:25.719084 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:26.730304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:26.730265 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:52:26.730928 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:26.730894 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 22:52:27.725595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:27.725553 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:52:27.725934 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:27.725898 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 22:52:28.733405 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:28.733348 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:52:28.733983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:28.733439 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 22:52:29.739221 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:29.735345 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:52:29.739221 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:29.735408 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 22:52:30.104860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:30.104763 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp"] Oct 29 22:52:30.105973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:30.105028 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" containerID="cri-o://98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232" gracePeriod=30 Oct 29 22:52:30.733180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:30.720581 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:52:30.733180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:30.720646 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 22:52:31.720636 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:31.720593 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:52:31.721271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:31.721233 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 22:52:32.724330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:32.724290 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:32.724827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:32.724793 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:32.728385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:32.726106 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 22:52:35.409051 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.409022 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:52:35.499646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.499612 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.499866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.499768 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.499866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.499803 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500025 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.499999 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500112 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500082 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.500191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500111 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500209 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500250 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500380 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500343 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500438 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500399 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo\") pod \"e9330a61-e457-47d4-bbbb-110325deecc3\" (UID: \"e9330a61-e457-47d4-bbbb-110325deecc3\") " Oct 29 22:52:35.500438 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500426 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.500547 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500457 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:35.500748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500723 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.500895 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.500800 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.501243 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.501219 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9330a61-e457-47d4-bbbb-110325deecc3-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.501477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.501384 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.501477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.501406 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.501477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.501423 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.501477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.501435 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.502843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.502808 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.503436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.503410 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:35.503757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.503737 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "kube-api-access-fjtwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:35.503820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.503763 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo" pod "e9330a61-e457-47d4-bbbb-110325deecc3" (UID: "e9330a61-e457-47d4-bbbb-110325deecc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:35.545786 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.545751 5073 generic.go:356] "Generic (PLEG): container finished" podID="e9330a61-e457-47d4-bbbb-110325deecc3" containerID="98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232" exitCode=0 Oct 29 22:52:35.545954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.545798 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" Oct 29 22:52:35.545954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.545829 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" event={"ID":"e9330a61-e457-47d4-bbbb-110325deecc3","Type":"ContainerDied","Data":"98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232"} Oct 29 22:52:35.545954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.545896 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp" event={"ID":"e9330a61-e457-47d4-bbbb-110325deecc3","Type":"ContainerDied","Data":"c39dcfe7fbc7249566fcd7d5908c4ea5380538cc8d2793327d287e84af5a2762"} Oct 29 22:52:35.545954 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.545919 5073 scope.go:122] "RemoveContainer" containerID="98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232" Oct 29 22:52:35.558027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.558007 5073 scope.go:122] "RemoveContainer" containerID="98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232" Oct 29 22:52:35.558389 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:52:35.558365 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232\": container with ID starting with 98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232 not found: ID does not exist" containerID="98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232" Oct 29 22:52:35.558441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.558400 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232"} err="failed to get container status \"98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232\": rpc error: code = NotFound desc = could not find container \"98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232\": container with ID starting with 98fa665e448cdfb6a6071ac6078ad97069e9cc5ecdaf0e4b9fd392e422dd4232 not found: ID does not exist" Oct 29 22:52:35.602893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.602851 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9330a61-e457-47d4-bbbb-110325deecc3-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.602893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.602887 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtwv\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-kube-api-access-fjtwv\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.602893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.602897 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9330a61-e457-47d4-bbbb-110325deecc3-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.603141 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.602905 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9330a61-e457-47d4-bbbb-110325deecc3-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.661875 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.661783 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp"] Oct 29 22:52:35.677632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:35.677596 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-hzmhp"] Oct 29 22:52:36.856565 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:36.856472 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" path="/var/lib/kubelet/pods/e9330a61-e457-47d4-bbbb-110325deecc3/volumes" Oct 29 22:52:39.273862 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:52:39.273813 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:52:40.172316 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.172280 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft"] Oct 29 22:52:40.173253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.173231 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" Oct 29 22:52:40.173253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.173252 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" Oct 29 22:52:40.173478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.173466 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9330a61-e457-47d4-bbbb-110325deecc3" containerName="istio-proxy" Oct 29 22:52:40.177640 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.177615 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.197442 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.197392 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft"] Oct 29 22:52:40.205260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.205203 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"default-dockercfg-n2599\"" Oct 29 22:52:40.205426 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.205284 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"rate-test-model-84304f-kserve-self-signed-certs\"" Oct 29 22:52:40.207224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.207196 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"openshift-service-ca.crt\"" Oct 29 22:52:40.207224 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.207199 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-84304f\"/\"kube-root-ca.crt\"" Oct 29 22:52:40.286505 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.286455 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.287119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.286647 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.287119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.286868 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.287119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.286930 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.287119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.286994 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.287326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.287129 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xd6n\" (UniqueName: \"kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389186 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389397 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389458 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389522 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389587 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xd6n\" (UniqueName: \"kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389711 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389677 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.389836 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.389809 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.390137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.390109 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.390137 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.390127 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.392363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.392332 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.392864 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.392843 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.420173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.418668 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xd6n\" (UniqueName: \"kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n\") pod \"rate-test-model-84304f-kserve-665c74ff49-8wmft\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.500447 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.500402 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:40.898105 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:52:40.898065 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2102a70f_da8f_4719_8ff4_d3481b9654fd.slice/crio-36c2b67e3f234169589606b94952cf05f57f7dccadba2445ae6c24db9f894b87 WatchSource:0}: Error finding container 36c2b67e3f234169589606b94952cf05f57f7dccadba2445ae6c24db9f894b87: Status 404 returned error can't find the container with id 36c2b67e3f234169589606b94952cf05f57f7dccadba2445ae6c24db9f894b87 Oct 29 22:52:40.898105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:40.898082 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft"] Oct 29 22:52:41.704548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:41.704499 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" event={"ID":"2102a70f-da8f-4719-8ff4-d3481b9654fd","Type":"ContainerStarted","Data":"c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21"} Oct 29 22:52:41.705212 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:41.705183 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" event={"ID":"2102a70f-da8f-4719-8ff4-d3481b9654fd","Type":"ContainerStarted","Data":"36c2b67e3f234169589606b94952cf05f57f7dccadba2445ae6c24db9f894b87"} Oct 29 22:52:41.755286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:41.755218 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" podStartSLOduration=1.755197903 podStartE2EDuration="1.755197903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:41.751628747 +0000 UTC m=+2277.067022829" watchObservedRunningTime="2026-10-29 22:52:41.755197903 +0000 UTC m=+2277.070591987" Oct 29 22:52:42.501635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:42.501599 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:50.501660 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:50.501610 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:50.515753 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:50.515716 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:50.785559 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:50.785474 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:52:51.986326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:51.986288 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf"] Oct 29 22:52:51.986797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:51.986478 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" containerID="cri-o://1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489" gracePeriod=30 Oct 29 22:52:57.162528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.161377 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54"] Oct 29 22:52:57.162528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.161623 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" containerID="cri-o://d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c" gracePeriod=30 Oct 29 22:52:57.196627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.195052 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft"] Oct 29 22:52:57.196627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.195648 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" containerID="cri-o://3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2" gracePeriod=30 Oct 29 22:52:57.208947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.208916 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd"] Oct 29 22:52:57.209502 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.209468 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" containerID="cri-o://2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8" gracePeriod=30 Oct 29 22:52:57.534092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.534028 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:52:57.672989 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.672956 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673004 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673049 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673088 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673128 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673265 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673338 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673453 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673487 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:57.673734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673558 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket\") pod \"e78b01c6-1ec2-4378-84c9-c57a6306b103\" (UID: \"e78b01c6-1ec2-4378-84c9-c57a6306b103\") " Oct 29 22:52:57.673734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673586 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:57.673841 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673726 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:57.673942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.673892 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:57.674114 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.674081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:57.674746 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.674721 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.674852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.674752 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.674852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.674767 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b01c6-1ec2-4378-84c9-c57a6306b103-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.674852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.674781 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.674852 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.674793 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.675874 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.675846 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "kube-api-access-cpn6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:57.675956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.675868 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:57.676110 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.676092 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:57.676232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.676214 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy" pod "e78b01c6-1ec2-4378-84c9-c57a6306b103" (UID: "e78b01c6-1ec2-4378-84c9-c57a6306b103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:57.776947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.776910 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.776947 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.776949 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpn6c\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-kube-api-access-cpn6c\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.777216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.776963 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.777216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.776979 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b01c6-1ec2-4378-84c9-c57a6306b103-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:52:57.822953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.822869 5073 generic.go:356] "Generic (PLEG): container finished" podID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerID="1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489" exitCode=0 Oct 29 22:52:57.822953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.822923 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" Oct 29 22:52:57.823186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.822956 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" event={"ID":"e78b01c6-1ec2-4378-84c9-c57a6306b103","Type":"ContainerDied","Data":"1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489"} Oct 29 22:52:57.823186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.823017 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf" event={"ID":"e78b01c6-1ec2-4378-84c9-c57a6306b103","Type":"ContainerDied","Data":"c8f136790e0be417634ac60a728cb72c9c35b404a289ecf06271f0ec24b0bc27"} Oct 29 22:52:57.823186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.823044 5073 scope.go:122] "RemoveContainer" containerID="1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489" Oct 29 22:52:57.834930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.834906 5073 scope.go:122] "RemoveContainer" containerID="1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489" Oct 29 22:52:57.835286 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:52:57.835266 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489\": container with ID starting with 1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489 not found: ID does not exist" containerID="1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489" Oct 29 22:52:57.835364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.835296 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489"} err="failed to get container status \"1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489\": rpc error: code = NotFound desc = could not find container \"1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489\": container with ID starting with 1246ea665411d0b54f74cc8bc15b1bd7c1736b2f59a6513d6daca6fff2b23489 not found: ID does not exist" Oct 29 22:52:57.881322 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.881286 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf"] Oct 29 22:52:57.891520 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:57.891490 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-rkzgf"] Oct 29 22:52:58.040342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:58.039018 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:58.040342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:58.039088 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:58.192825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:58.192739 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:58.192825 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:58.192796 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:58.839232 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:58.839198 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" path="/var/lib/kubelet/pods/e78b01c6-1ec2-4378-84c9-c57a6306b103/volumes" Oct 29 22:52:59.183339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:59.183256 5073 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:59.183498 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:52:59.183347 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:02.571477 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.571446 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:53:02.581430 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.581396 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:53:02.618517 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.618490 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:53:02.654974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.654940 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.654974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.654994 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.655305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.655044 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.655305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.655082 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.655305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.655120 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.655305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.655193 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.655305 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.655221 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.658868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.657861 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.658868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.655269 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.658868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.658751 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.658868 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.658824 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.659313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.659032 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.659620 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.659591 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.660262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660233 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.660373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660323 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.660373 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660355 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.660493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660387 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.660493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660418 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.660493 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660460 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.660667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660531 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.660667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660624 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.660667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660661 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.660828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660728 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.660828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660763 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.660828 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660806 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.660994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660844 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.660994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660876 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.660994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660911 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert\") pod \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\" (UID: \"9b6d5106-733c-4ebd-af8c-c88968b0f0cf\") " Oct 29 22:53:02.660994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.660964 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert\") pod \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\" (UID: \"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86\") " Oct 29 22:53:02.661297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.661001 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.661297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.661111 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.661297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.661204 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs\") pod \"d41cca39-790e-4f63-827b-c3cd3be0fd37\" (UID: \"d41cca39-790e-4f63-827b-c3cd3be0fd37\") " Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.661702 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.661930 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662183 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662207 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662223 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662238 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662253 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662268 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662281 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.662594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662395 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.663180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662628 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.663180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662691 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.663180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662779 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.663180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.662925 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.663180 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.663069 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.663450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.663298 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.663450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.663301 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.663631 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.663607 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.666227 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.665899 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.666227 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.666138 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.666227 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.666184 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.668513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.668490 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.669049 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.669021 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "kube-api-access-m8wst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.669504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.669481 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy" pod "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" (UID: "9f28fc48-9277-42f8-90d7-b1ef9d6a3b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.669631 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.669613 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "kube-api-access-jzbb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.669757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.669734 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.669757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.669746 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy" pod "d41cca39-790e-4f63-827b-c3cd3be0fd37" (UID: "d41cca39-790e-4f63-827b-c3cd3be0fd37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.670104 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.669738 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.671579 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.671517 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck" pod "9b6d5106-733c-4ebd-af8c-c88968b0f0cf" (UID: "9b6d5106-733c-4ebd-af8c-c88968b0f0cf"). InnerVolumeSpecName "kube-api-access-qvkck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.763818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763739 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.763818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763770 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.763818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763786 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wst\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-kube-api-access-m8wst\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.763818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763799 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvkck\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-kube-api-access-qvkck\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.763818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763808 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41cca39-790e-4f63-827b-c3cd3be0fd37-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.763818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763817 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763825 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763833 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763841 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763849 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763860 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzbb9\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-kube-api-access-jzbb9\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763873 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763881 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763889 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763896 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763904 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763912 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763918 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41cca39-790e-4f63-827b-c3cd3be0fd37-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763925 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d5106-733c-4ebd-af8c-c88968b0f0cf-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.764121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.763936 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.863669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.863636 5073 generic.go:356] "Generic (PLEG): container finished" podID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerID="2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8" exitCode=0 Oct 29 22:53:02.863894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.863696 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" Oct 29 22:53:02.863894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.863720 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" event={"ID":"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86","Type":"ContainerDied","Data":"2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8"} Oct 29 22:53:02.863894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.863781 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd" event={"ID":"9f28fc48-9277-42f8-90d7-b1ef9d6a3b86","Type":"ContainerDied","Data":"7bf9ab5e32e147553aa286a5b221ddbaf1e33432a1211d1b3198c3be80f3b85e"} Oct 29 22:53:02.863894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.863842 5073 scope.go:122] "RemoveContainer" containerID="2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8" Oct 29 22:53:02.868562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.868537 5073 generic.go:356] "Generic (PLEG): container finished" podID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerID="3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2" exitCode=0 Oct 29 22:53:02.868669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.868606 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" Oct 29 22:53:02.868669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.868646 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" event={"ID":"9b6d5106-733c-4ebd-af8c-c88968b0f0cf","Type":"ContainerDied","Data":"3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2"} Oct 29 22:53:02.868780 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.868696 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft" event={"ID":"9b6d5106-733c-4ebd-af8c-c88968b0f0cf","Type":"ContainerDied","Data":"0f6e2227c45cc2262afedc308794c8a87311b8d10cca6bfde9a6962c010e6bbe"} Oct 29 22:53:02.870683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.870656 5073 generic.go:356] "Generic (PLEG): container finished" podID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerID="d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c" exitCode=0 Oct 29 22:53:02.870793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.870764 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" event={"ID":"d41cca39-790e-4f63-827b-c3cd3be0fd37","Type":"ContainerDied","Data":"d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c"} Oct 29 22:53:02.870857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.870802 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" event={"ID":"d41cca39-790e-4f63-827b-c3cd3be0fd37","Type":"ContainerDied","Data":"f2d2b860a84e7f5df96978b7f6c1aa43ac851a4b6f575423d72c8ac239c9ef2d"} Oct 29 22:53:02.870913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.870866 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54" Oct 29 22:53:02.879901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.879874 5073 scope.go:122] "RemoveContainer" containerID="2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8" Oct 29 22:53:02.880187 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:53:02.880168 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8\": container with ID starting with 2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8 not found: ID does not exist" containerID="2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8" Oct 29 22:53:02.880251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.880198 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8"} err="failed to get container status \"2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8\": rpc error: code = NotFound desc = could not find container \"2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8\": container with ID starting with 2bfebac765eddf58fa57e35b18e3dd86907c8c616f6859829dac20301aa0d5f8 not found: ID does not exist" Oct 29 22:53:02.880251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.880216 5073 scope.go:122] "RemoveContainer" containerID="3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2" Oct 29 22:53:02.890326 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.890308 5073 scope.go:122] "RemoveContainer" containerID="3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2" Oct 29 22:53:02.890601 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:53:02.890572 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2\": container with ID starting with 3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2 not found: ID does not exist" containerID="3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2" Oct 29 22:53:02.890664 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.890608 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2"} err="failed to get container status \"3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2\": rpc error: code = NotFound desc = could not find container \"3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2\": container with ID starting with 3a1926f7caecd9054017cbfc5ac82c799427462dd9a24b204575c4510094e0b2 not found: ID does not exist" Oct 29 22:53:02.890664 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.890625 5073 scope.go:122] "RemoveContainer" containerID="d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c" Oct 29 22:53:02.900279 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.900257 5073 scope.go:122] "RemoveContainer" containerID="d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c" Oct 29 22:53:02.900558 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:53:02.900540 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c\": container with ID starting with d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c not found: ID does not exist" containerID="d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c" Oct 29 22:53:02.900610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.900567 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c"} err="failed to get container status \"d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c\": rpc error: code = NotFound desc = could not find container \"d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c\": container with ID starting with d61aeec79616cfc18ef3da9476c4fca5e6b759080d6cd3738a836f5a372e7a7c not found: ID does not exist" Oct 29 22:53:02.921986 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.921959 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft"] Oct 29 22:53:02.926139 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.926109 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-jpmft"] Oct 29 22:53:02.966394 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.966360 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54"] Oct 29 22:53:02.970077 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:02.970048 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-q4h54"] Oct 29 22:53:03.009894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:03.009839 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd"] Oct 29 22:53:03.013571 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:03.013547 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-nbwcd"] Oct 29 22:53:04.841391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:04.841357 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" path="/var/lib/kubelet/pods/9b6d5106-733c-4ebd-af8c-c88968b0f0cf/volumes" Oct 29 22:53:04.841933 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:04.841917 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" path="/var/lib/kubelet/pods/9f28fc48-9277-42f8-90d7-b1ef9d6a3b86/volumes" Oct 29 22:53:04.842392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:04.842378 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" path="/var/lib/kubelet/pods/d41cca39-790e-4f63-827b-c3cd3be0fd37/volumes" Oct 29 22:53:06.858731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:06.858697 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:53:09.273850 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:53:09.273768 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:53:22.902030 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.901996 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp"] Oct 29 22:53:22.902792 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902773 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902790 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902825 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902832 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902849 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902854 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902876 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" Oct 29 22:53:22.902942 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.902882 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" Oct 29 22:53:22.903296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.903021 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41cca39-790e-4f63-827b-c3cd3be0fd37" containerName="istio-proxy" Oct 29 22:53:22.903296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.903111 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78b01c6-1ec2-4378-84c9-c57a6306b103" containerName="istio-proxy" Oct 29 22:53:22.903296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.903125 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f28fc48-9277-42f8-90d7-b1ef9d6a3b86" containerName="istio-proxy" Oct 29 22:53:22.903296 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.903138 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6d5106-733c-4ebd-af8c-c88968b0f0cf" containerName="istio-proxy" Oct 29 22:53:22.907086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.907069 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.938812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.938776 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-dbb507-openshift-default-dockercfg-54prz\"" Oct 29 22:53:22.945134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.945091 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp"] Oct 29 22:53:22.970387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970343 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970464 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970498 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970617 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970544 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxts6\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970613 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970661 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970637 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970726 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970694 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970748 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:22.970830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:22.970814 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072356 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072480 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072533 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072552 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072703 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072670 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxts6\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072760 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072743 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072808 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072776 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072804 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072823 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.072924 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072831 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.073048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072935 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.073048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.072964 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.073192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.073143 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.073594 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.073577 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.074969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.074949 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.075407 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.075388 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.094680 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.094649 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.095219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.095200 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxts6\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.222621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.222576 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:23.551866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.551832 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp"] Oct 29 22:53:23.552125 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:53:23.552105 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306897f3_fa2f_4117_9b46_daaf3045dfe9.slice/crio-99e9d11a3c996b66a2ca4c0eefea753395a169c06f52769de5ac099e0ca64197 WatchSource:0}: Error finding container 99e9d11a3c996b66a2ca4c0eefea753395a169c06f52769de5ac099e0ca64197: Status 404 returned error can't find the container with id 99e9d11a3c996b66a2ca4c0eefea753395a169c06f52769de5ac099e0ca64197 Oct 29 22:53:23.555087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.555052 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:53:23.555185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.555124 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:53:23.555185 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:23.555170 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:53:24.062626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:24.062589 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" event={"ID":"306897f3-fa2f-4117-9b46-daaf3045dfe9","Type":"ContainerStarted","Data":"b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a"} Oct 29 22:53:24.063119 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:24.062639 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" event={"ID":"306897f3-fa2f-4117-9b46-daaf3045dfe9","Type":"ContainerStarted","Data":"99e9d11a3c996b66a2ca4c0eefea753395a169c06f52769de5ac099e0ca64197"} Oct 29 22:53:24.111360 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:24.111312 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podStartSLOduration=2.111298427 podStartE2EDuration="2.111298427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:24.110497395 +0000 UTC m=+2319.425891475" watchObservedRunningTime="2026-10-29 22:53:24.111298427 +0000 UTC m=+2319.426692507" Oct 29 22:53:24.222978 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:24.222936 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:24.225195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:24.225160 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:53:24.225319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:24.225212 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 22:53:25.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:25.230258 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:53:25.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:25.230325 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 22:53:26.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:26.230603 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:53:26.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:26.230672 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 22:53:27.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:27.227558 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:53:27.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:27.227638 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 22:53:28.231722 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:28.231683 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:53:28.232366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:28.232333 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 22:53:29.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:29.230656 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:53:29.231178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:29.230734 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 22:53:30.420573 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:30.420535 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:30.421857 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:30.421831 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:30.424064 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:30.424041 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 22:53:39.256333 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:53:39.256286 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:53:40.564700 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:40.564662 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg"] Oct 29 22:53:40.565255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:40.565012 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" containerID="cri-o://1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7" gracePeriod=30 Oct 29 22:53:45.885492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:45.885456 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:53:46.019167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019115 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019216 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019246 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019340 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019282 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019350 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019379 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019400 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.019465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019407 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019689 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019668 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019737 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs\") pod \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\" (UID: \"3ad4974f-42cf-49da-9f4a-a39415cfcb3f\") " Oct 29 22:53:46.019764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019741 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:46.020020 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.019991 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.020255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020225 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.020377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020334 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.020612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020511 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.020612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020529 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.020612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020538 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.020612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020546 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.020612 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.020555 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.022304 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.022278 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:46.022413 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.022349 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "kube-api-access-5mp68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:46.022543 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.022521 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:46.022604 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.022521 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy" pod "3ad4974f-42cf-49da-9f4a-a39415cfcb3f" (UID: "3ad4974f-42cf-49da-9f4a-a39415cfcb3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:46.121992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.121894 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.121992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.121929 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mp68\" (UniqueName: \"kubernetes.io/projected/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-kube-api-access-5mp68\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.121992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.121939 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.121992 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.121949 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4974f-42cf-49da-9f4a-a39415cfcb3f-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:53:46.276876 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.276842 5073 generic.go:356] "Generic (PLEG): container finished" podID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerID="1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7" exitCode=0 Oct 29 22:53:46.277072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.276902 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" Oct 29 22:53:46.277072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.276929 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" event={"ID":"3ad4974f-42cf-49da-9f4a-a39415cfcb3f","Type":"ContainerDied","Data":"1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7"} Oct 29 22:53:46.277072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.277000 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg" event={"ID":"3ad4974f-42cf-49da-9f4a-a39415cfcb3f","Type":"ContainerDied","Data":"5a24de6af9114b06de4fa3c6f62b1267116fcf6c70dd2ffbc0e0241632844541"} Oct 29 22:53:46.277072 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.277031 5073 scope.go:122] "RemoveContainer" containerID="1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7" Oct 29 22:53:46.289519 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.289499 5073 scope.go:122] "RemoveContainer" containerID="1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7" Oct 29 22:53:46.289842 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:53:46.289823 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7\": container with ID starting with 1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7 not found: ID does not exist" containerID="1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7" Oct 29 22:53:46.289933 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.289852 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7"} err="failed to get container status \"1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7\": rpc error: code = NotFound desc = could not find container \"1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7\": container with ID starting with 1662310448efe7e4a934962ef99fc0c592ef8b5073657dece37ea03fda3a5da7 not found: ID does not exist" Oct 29 22:53:46.343041 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.342991 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg"] Oct 29 22:53:46.349723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.349693 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8ebc60-openshift-default-78b65df4d9-8xlkg"] Oct 29 22:53:46.840015 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:53:46.839971 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" path="/var/lib/kubelet/pods/3ad4974f-42cf-49da-9f4a-a39415cfcb3f/volumes" Oct 29 22:54:09.257092 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:09.257044 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:54:16.663975 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:16.663940 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s"] Oct 29 22:54:16.664500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:16.664186 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" containerID="cri-o://d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf" gracePeriod=30 Oct 29 22:54:16.678633 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:16.678595 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr"] Oct 29 22:54:16.678899 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:16.678865 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" containerID="cri-o://6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3" gracePeriod=30 Oct 29 22:54:21.146239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:21.146197 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:54:21.146654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:21.146258 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:54:21.947510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:21.947481 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:54:21.947632 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:21.947549 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 29 22:54:21.979704 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:21.979679 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:54:22.127994 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.127958 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128071 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128133 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128194 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128192 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128234 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128309 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128367 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128414 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.128555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128292 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.128555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128497 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket\") pod \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\" (UID: \"eaf58a7a-5a11-4f93-a8ba-cd8982811b39\") " Oct 29 22:54:22.129551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.128809 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.129551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.129224 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:22.129551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.129286 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.129551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.129409 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.129551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.129506 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.130452 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.129325 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.130835 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.130808 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.131226 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.131205 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:54:22.131603 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.131583 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "kube-api-access-4947t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:22.131777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.131759 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token" pod "eaf58a7a-5a11-4f93-a8ba-cd8982811b39" (UID: "eaf58a7a-5a11-4f93-a8ba-cd8982811b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233476 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4947t\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-kube-api-access-4947t\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233515 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233526 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233535 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233544 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233553 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.235270 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.233566 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf58a7a-5a11-4f93-a8ba-cd8982811b39-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.378969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.378940 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:54:22.544930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.544851 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.544930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.544890 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.544930 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.544911 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.544940 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545027 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545101 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545162 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545240 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545253 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.545619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545285 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket\") pod \"56eb11cf-7c12-4856-8605-88463865fcd4\" (UID: \"56eb11cf-7c12-4856-8605-88463865fcd4\") " Oct 29 22:54:22.545619 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545494 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.545944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545922 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.546201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.545429 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.546201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.546176 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:22.546400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.546273 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.546400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.546298 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.546400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.546313 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.546400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.546327 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.547662 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.547637 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:22.547793 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.547775 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:22.547850 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.547830 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:54:22.547889 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.547841 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6" pod "56eb11cf-7c12-4856-8605-88463865fcd4" (UID: "56eb11cf-7c12-4856-8605-88463865fcd4"). InnerVolumeSpecName "kube-api-access-2nml6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:22.559325 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.559299 5073 generic.go:356] "Generic (PLEG): container finished" podID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerID="d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf" exitCode=0 Oct 29 22:54:22.559433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.559344 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" Oct 29 22:54:22.559433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.559375 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" event={"ID":"eaf58a7a-5a11-4f93-a8ba-cd8982811b39","Type":"ContainerDied","Data":"d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf"} Oct 29 22:54:22.559433 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.559428 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s" event={"ID":"eaf58a7a-5a11-4f93-a8ba-cd8982811b39","Type":"ContainerDied","Data":"9741dabd74202671f166f9985717fae82cb8ebe7ec262a93395712fdecad5c02"} Oct 29 22:54:22.559585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.559450 5073 scope.go:122] "RemoveContainer" containerID="d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf" Oct 29 22:54:22.562082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.562050 5073 generic.go:356] "Generic (PLEG): container finished" podID="56eb11cf-7c12-4856-8605-88463865fcd4" containerID="6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3" exitCode=0 Oct 29 22:54:22.562195 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.562120 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" event={"ID":"56eb11cf-7c12-4856-8605-88463865fcd4","Type":"ContainerDied","Data":"6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3"} Oct 29 22:54:22.562244 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.562199 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" event={"ID":"56eb11cf-7c12-4856-8605-88463865fcd4","Type":"ContainerDied","Data":"e850d4f2ff87c599f837395a3b9c198e91225b5bd7bd80eafea792a0a916bda9"} Oct 29 22:54:22.562290 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.562265 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr" Oct 29 22:54:22.572528 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.572511 5073 scope.go:122] "RemoveContainer" containerID="d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf" Oct 29 22:54:22.572805 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:22.572784 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf\": container with ID starting with d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf not found: ID does not exist" containerID="d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf" Oct 29 22:54:22.572854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.572813 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf"} err="failed to get container status \"d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf\": rpc error: code = NotFound desc = could not find container \"d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf\": container with ID starting with d3abd42b5215c7cfcc3a499fe38ac4268b03806eba38aaa06a43bd04f831e2bf not found: ID does not exist" Oct 29 22:54:22.572854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.572830 5073 scope.go:122] "RemoveContainer" containerID="6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3" Oct 29 22:54:22.582091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.582073 5073 scope.go:122] "RemoveContainer" containerID="a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805" Oct 29 22:54:22.591705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.591687 5073 scope.go:122] "RemoveContainer" containerID="6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3" Oct 29 22:54:22.591977 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:22.591953 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3\": container with ID starting with 6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3 not found: ID does not exist" containerID="6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3" Oct 29 22:54:22.592037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.591985 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3"} err="failed to get container status \"6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3\": rpc error: code = NotFound desc = could not find container \"6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3\": container with ID starting with 6b8b7b9122b8147a1b09576cf31c5971c3e8dbe93e21c851df28779a7fccc4f3 not found: ID does not exist" Oct 29 22:54:22.592037 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.592005 5073 scope.go:122] "RemoveContainer" containerID="a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805" Oct 29 22:54:22.592266 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:22.592248 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805\": container with ID starting with a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805 not found: ID does not exist" containerID="a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805" Oct 29 22:54:22.592344 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.592274 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805"} err="failed to get container status \"a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805\": rpc error: code = NotFound desc = could not find container \"a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805\": container with ID starting with a1fad7c078451bf7271ade729285167a4dec73d779b77d90866e85a7dad86805 not found: ID does not exist" Oct 29 22:54:22.620084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.620007 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s"] Oct 29 22:54:22.625820 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.625799 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-nmx8s"] Oct 29 22:54:22.647869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.647849 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb11cf-7c12-4856-8605-88463865fcd4-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.647869 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.647871 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb11cf-7c12-4856-8605-88463865fcd4-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.648036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.647880 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb11cf-7c12-4856-8605-88463865fcd4-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.648036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.647888 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nml6\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-kube-api-access-2nml6\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.648036 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.647897 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb11cf-7c12-4856-8605-88463865fcd4-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:22.660849 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.660828 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr"] Oct 29 22:54:22.663843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.663823 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-q5zwr"] Oct 29 22:54:22.842764 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.842689 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" path="/var/lib/kubelet/pods/56eb11cf-7c12-4856-8605-88463865fcd4/volumes" Oct 29 22:54:22.843262 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:22.843248 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" path="/var/lib/kubelet/pods/eaf58a7a-5a11-4f93-a8ba-cd8982811b39/volumes" Oct 29 22:54:24.181728 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.181645 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw"] Oct 29 22:54:24.182456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182436 5073 cpu_manager.go:414] "containerMap: removing container" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" Oct 29 22:54:24.182569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182457 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" Oct 29 22:54:24.182569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182518 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" Oct 29 22:54:24.182569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182527 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" Oct 29 22:54:24.182569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182554 5073 cpu_manager.go:414] "containerMap: removing container" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" Oct 29 22:54:24.182569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182562 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" Oct 29 22:54:24.182832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182579 5073 cpu_manager.go:414] "containerMap: removing container" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" Oct 29 22:54:24.182832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182587 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" Oct 29 22:54:24.182832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182736 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" Oct 29 22:54:24.182832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182754 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf58a7a-5a11-4f93-a8ba-cd8982811b39" containerName="istio-proxy" Oct 29 22:54:24.182832 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.182773 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad4974f-42cf-49da-9f4a-a39415cfcb3f" containerName="istio-proxy" Oct 29 22:54:24.188046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.188024 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.207222 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.207196 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw"] Oct 29 22:54:24.213522 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.213501 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-329f7b-serving-cert\"" Oct 29 22:54:24.378598 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.368991 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.378950 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.378912 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.379047 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.378995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq47s\" (UniqueName: \"kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.383335 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.379309 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.480955 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.480921 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.481184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.480990 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq47s\" (UniqueName: \"kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.481184 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.481124 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.481289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.481274 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.481441 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:24.481420 5073 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-serving-cert: secret "maas-api-e2e-shared-a-w1-329f7b-serving-cert" not found Oct 29 22:54:24.481518 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:24.481504 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls podName:351d38c6-6670-4848-be61-2b55171cf466 nodeName:}" failed. No retries permitted until 2026-10-29 22:54:24.981483099 +0000 UTC m=+2380.296877165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls") pod "maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" (UID: "351d38c6-6670-4848-be61-2b55171cf466") : secret "maas-api-e2e-shared-a-w1-329f7b-serving-cert" not found Oct 29 22:54:24.481686 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.481667 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.484022 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.483999 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.504655 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.504628 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq47s\" (UniqueName: \"kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.986923 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.986887 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:24.989683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:24.989651 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:25.107588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:25.107549 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:25.462302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:25.462264 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw"] Oct 29 22:54:25.462933 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:54:25.462833 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351d38c6_6670_4848_be61_2b55171cf466.slice/crio-22c92e2fd14f80b89e1cf847f3d358ae5d5c42c510cd9f4563ed44e0ef73a10f WatchSource:0}: Error finding container 22c92e2fd14f80b89e1cf847f3d358ae5d5c42c510cd9f4563ed44e0ef73a10f: Status 404 returned error can't find the container with id 22c92e2fd14f80b89e1cf847f3d358ae5d5c42c510cd9f4563ed44e0ef73a10f Oct 29 22:54:25.464512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:25.464491 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:54:25.586586 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:25.586560 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" event={"ID":"351d38c6-6670-4848-be61-2b55171cf466","Type":"ContainerStarted","Data":"22c92e2fd14f80b89e1cf847f3d358ae5d5c42c510cd9f4563ed44e0ef73a10f"} Oct 29 22:54:26.595107 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:26.595064 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" event={"ID":"351d38c6-6670-4848-be61-2b55171cf466","Type":"ContainerStarted","Data":"7437314622cfe477039ed4a1a5126056c2f9a608046acb9492322b6481ea2d7a"} Oct 29 22:54:26.595501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:26.595124 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:26.641674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:26.641622 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" podStartSLOduration=2.310058174 podStartE2EDuration="2.641605326s" totalImagesPullingTime="331.547152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:26.637278753 +0000 UTC m=+2381.952672834" watchObservedRunningTime="2026-10-29 22:54:26.641605326 +0000 UTC m=+2381.956999407" Oct 29 22:54:32.607713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:32.607683 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 22:54:33.488073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.488041 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj"] Oct 29 22:54:33.488860 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.488839 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eb11cf-7c12-4856-8605-88463865fcd4" containerName="istio-proxy" Oct 29 22:54:33.492563 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.492541 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.512944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.512915 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj"] Oct 29 22:54:33.522375 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.522349 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"kube-root-ca.crt\"" Oct 29 22:54:33.522506 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.522416 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"default-dockercfg-knfgv\"" Oct 29 22:54:33.523230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.523210 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"rate-test-model-437981-kserve-self-signed-certs\"" Oct 29 22:54:33.523339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.523210 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"openshift-service-ca.crt\"" Oct 29 22:54:33.610389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.610357 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.610751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.610525 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.610751 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.610728 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8b8\" (UniqueName: \"kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.610826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.610771 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.610929 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.610905 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.611015 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.610980 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712425 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712496 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712643 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712528 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712647 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712842 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712814 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8b8\" (UniqueName: \"kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712852 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.712900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.712863 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.713363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.713195 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.713363 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.713269 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.715357 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.715330 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.715952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.715929 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.736403 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.736330 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8b8\" (UniqueName: \"kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8\") pod \"rate-test-model-437981-kserve-857f5cfbfc-2lkgj\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:33.814738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:33.814415 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:34.156457 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:34.156408 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj"] Oct 29 22:54:34.160056 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:54:34.160024 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ac4a115_a240_459c_9853_d9d11faa6df7.slice/crio-c2a18c8891ba3fcdd7b1e180c2628f0f02cf968df776d6ba696df567a985cf2c WatchSource:0}: Error finding container c2a18c8891ba3fcdd7b1e180c2628f0f02cf968df776d6ba696df567a985cf2c: Status 404 returned error can't find the container with id c2a18c8891ba3fcdd7b1e180c2628f0f02cf968df776d6ba696df567a985cf2c Oct 29 22:54:34.652781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:34.652732 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" event={"ID":"4ac4a115-a240-459c-9853-d9d11faa6df7","Type":"ContainerStarted","Data":"c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23"} Oct 29 22:54:34.653187 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:34.652789 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" event={"ID":"4ac4a115-a240-459c-9853-d9d11faa6df7","Type":"ContainerStarted","Data":"c2a18c8891ba3fcdd7b1e180c2628f0f02cf968df776d6ba696df567a985cf2c"} Oct 29 22:54:34.696717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:34.696668 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" podStartSLOduration=1.696654967 podStartE2EDuration="1.696654967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:34.694315622 +0000 UTC m=+2390.009709702" watchObservedRunningTime="2026-10-29 22:54:34.696654967 +0000 UTC m=+2390.012049046" Oct 29 22:54:34.816136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:34.816099 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:39.270252 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:39.270210 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:54:43.815391 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:43.815359 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:43.828843 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:43.828820 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:43.891206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:43.881816 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:54:44.748400 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:44.748362 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:54:45.766777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:45.766746 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:54:45.770361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:45.770340 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:54:45.772983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:45.772959 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:54:45.779209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:45.779185 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:54:49.397727 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.397684 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9023-ccd781"] Oct 29 22:54:49.403460 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.403435 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:49.417216 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.417192 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-ccd781"] Oct 29 22:54:49.419531 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.419497 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c5k5w\"" Oct 29 22:54:49.530681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.530640 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfw9\" (UniqueName: \"kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9\") pod \"test-curl-9023-ccd781\" (UID: \"53d566b6-9567-4e79-a0eb-9777b7b7859a\") " pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:49.632092 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.632053 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfw9\" (UniqueName: \"kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9\") pod \"test-curl-9023-ccd781\" (UID: \"53d566b6-9567-4e79-a0eb-9777b7b7859a\") " pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:49.654178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.654082 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfw9\" (UniqueName: \"kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9\") pod \"test-curl-9023-ccd781\" (UID: \"53d566b6-9567-4e79-a0eb-9777b7b7859a\") " pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:49.720459 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:49.720434 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:50.037958 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:54:50.037928 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53d566b6_9567_4e79_a0eb_9777b7b7859a.slice/crio-5d2510af2033aaddd8967225cb5a5bcd09db5433b498bf4bfd8894a62bfb015c WatchSource:0}: Error finding container 5d2510af2033aaddd8967225cb5a5bcd09db5433b498bf4bfd8894a62bfb015c: Status 404 returned error can't find the container with id 5d2510af2033aaddd8967225cb5a5bcd09db5433b498bf4bfd8894a62bfb015c Oct 29 22:54:50.039074 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:50.039047 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-ccd781"] Oct 29 22:54:50.841979 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:50.841943 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-ccd781" event={"ID":"53d566b6-9567-4e79-a0eb-9777b7b7859a","Type":"ContainerStarted","Data":"7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7"} Oct 29 22:54:50.842389 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:50.841987 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-ccd781" event={"ID":"53d566b6-9567-4e79-a0eb-9777b7b7859a","Type":"ContainerStarted","Data":"5d2510af2033aaddd8967225cb5a5bcd09db5433b498bf4bfd8894a62bfb015c"} Oct 29 22:54:50.881318 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:50.881270 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9023-ccd781" podStartSLOduration=1.881255954 podStartE2EDuration="1.881255954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:50.879621313 +0000 UTC m=+2406.195015394" watchObservedRunningTime="2026-10-29 22:54:50.881255954 +0000 UTC m=+2406.196650034" Oct 29 22:54:51.742752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.742677 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9023-ccd781"] Oct 29 22:54:51.747093 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.747061 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9023-ccd781"] Oct 29 22:54:51.901815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.901781 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9023-60de05"] Oct 29 22:54:51.902467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.902449 5073 cpu_manager.go:414] "containerMap: removing container" podUID="53d566b6-9567-4e79-a0eb-9777b7b7859a" containerName="test-curl-9023-ccd781" Oct 29 22:54:51.902512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.902469 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d566b6-9567-4e79-a0eb-9777b7b7859a" containerName="test-curl-9023-ccd781" Oct 29 22:54:51.902615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.902605 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d566b6-9567-4e79-a0eb-9777b7b7859a" containerName="test-curl-9023-ccd781" Oct 29 22:54:51.906240 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.906220 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:51.919359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:51.919338 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-60de05"] Oct 29 22:54:52.067401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.067326 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncd6\" (UniqueName: \"kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6\") pod \"test-curl-9023-60de05\" (UID: \"a0bca0c5-5002-4903-9408-db12765bf2fd\") " pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:52.169035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.169006 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncd6\" (UniqueName: \"kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6\") pod \"test-curl-9023-60de05\" (UID: \"a0bca0c5-5002-4903-9408-db12765bf2fd\") " pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:52.190463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.190438 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncd6\" (UniqueName: \"kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6\") pod \"test-curl-9023-60de05\" (UID: \"a0bca0c5-5002-4903-9408-db12765bf2fd\") " pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:52.221969 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.221945 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:52.523401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.523370 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9023-60de05"] Oct 29 22:54:52.523525 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:54:52.523435 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0bca0c5_5002_4903_9408_db12765bf2fd.slice/crio-c474bc2d8dbd755b68a2a73e41cd498a9f7dd7fe5ba41033ebd6d464474de389 WatchSource:0}: Error finding container c474bc2d8dbd755b68a2a73e41cd498a9f7dd7fe5ba41033ebd6d464474de389: Status 404 returned error can't find the container with id c474bc2d8dbd755b68a2a73e41cd498a9f7dd7fe5ba41033ebd6d464474de389 Oct 29 22:54:52.854960 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.854877 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-60de05" event={"ID":"a0bca0c5-5002-4903-9408-db12765bf2fd","Type":"ContainerStarted","Data":"e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30"} Oct 29 22:54:52.854960 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.854916 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9023-60de05" event={"ID":"a0bca0c5-5002-4903-9408-db12765bf2fd","Type":"ContainerStarted","Data":"c474bc2d8dbd755b68a2a73e41cd498a9f7dd7fe5ba41033ebd6d464474de389"} Oct 29 22:54:52.855225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.855143 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9023-ccd781" podUID="53d566b6-9567-4e79-a0eb-9777b7b7859a" containerName="test-curl-9023-ccd781" containerID="cri-o://7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7" gracePeriod=2 Oct 29 22:54:52.898097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:52.898053 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9023-60de05" podStartSLOduration=1.8980397390000001 podStartE2EDuration="1.898039739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:52.894821116 +0000 UTC m=+2408.210215193" watchObservedRunningTime="2026-10-29 22:54:52.898039739 +0000 UTC m=+2408.213433819" Oct 29 22:54:53.683630 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:53.683573 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9023-60de05"] Oct 29 22:54:53.687330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:53.687302 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9023-60de05"] Oct 29 22:54:54.869201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:54.869125 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9023-60de05" podUID="a0bca0c5-5002-4903-9408-db12765bf2fd" containerName="test-curl-9023-60de05" containerID="cri-o://e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30" gracePeriod=2 Oct 29 22:54:55.086510 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.086487 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:55.208871 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.208849 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9\" (UniqueName: \"kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9\") pod \"53d566b6-9567-4e79-a0eb-9777b7b7859a\" (UID: \"53d566b6-9567-4e79-a0eb-9777b7b7859a\") " Oct 29 22:54:55.211038 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.211013 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9" pod "53d566b6-9567-4e79-a0eb-9777b7b7859a" (UID: "53d566b6-9567-4e79-a0eb-9777b7b7859a"). InnerVolumeSpecName "kube-api-access-4sfw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:55.311048 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.311015 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfw9\" (UniqueName: \"kubernetes.io/projected/53d566b6-9567-4e79-a0eb-9777b7b7859a-kube-api-access-4sfw9\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:55.877188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.877133 5073 generic.go:356] "Generic (PLEG): container finished" podID="53d566b6-9567-4e79-a0eb-9777b7b7859a" containerID="7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7" exitCode=137 Oct 29 22:54:55.877646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.877197 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:55.877646 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.877242 5073 scope.go:122] "RemoveContainer" containerID="7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7" Oct 29 22:54:55.890478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.890461 5073 scope.go:122] "RemoveContainer" containerID="7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7" Oct 29 22:54:55.890745 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:55.890723 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7\": container with ID starting with 7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7 not found: ID does not exist" containerID="7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7" Oct 29 22:54:55.890787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:55.890755 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7"} err="failed to get container status \"7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7\": rpc error: code = NotFound desc = could not find container \"7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7\": container with ID starting with 7a55fb9e387cf28935d5029625f8a4f59296233bc20fd2634ae3d587eb5d80b7 not found: ID does not exist" Oct 29 22:54:55.902650 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:55.902625 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9023-ccd781\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="53d566b6-9567-4e79-a0eb-9777b7b7859a" pod="opendatahub/test-curl-9023-ccd781" Oct 29 22:54:56.839757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:56.839720 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d566b6-9567-4e79-a0eb-9777b7b7859a" path="/var/lib/kubelet/pods/53d566b6-9567-4e79-a0eb-9777b7b7859a/volumes" Oct 29 22:54:57.092225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.092117 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:57.236191 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.236161 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6\" (UniqueName: \"kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6\") pod \"a0bca0c5-5002-4903-9408-db12765bf2fd\" (UID: \"a0bca0c5-5002-4903-9408-db12765bf2fd\") " Oct 29 22:54:57.238467 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.238438 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6" pod "a0bca0c5-5002-4903-9408-db12765bf2fd" (UID: "a0bca0c5-5002-4903-9408-db12765bf2fd"). InnerVolumeSpecName "kube-api-access-jncd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:57.338192 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.338116 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jncd6\" (UniqueName: \"kubernetes.io/projected/a0bca0c5-5002-4903-9408-db12765bf2fd-kube-api-access-jncd6\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:54:57.891965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.891925 5073 generic.go:356] "Generic (PLEG): container finished" podID="a0bca0c5-5002-4903-9408-db12765bf2fd" containerID="e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30" exitCode=137 Oct 29 22:54:57.891965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.891966 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:57.892239 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.892021 5073 scope.go:122] "RemoveContainer" containerID="e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30" Oct 29 22:54:57.906917 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.906888 5073 scope.go:122] "RemoveContainer" containerID="e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30" Oct 29 22:54:57.907357 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:57.907331 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30\": container with ID starting with e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30 not found: ID does not exist" containerID="e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30" Oct 29 22:54:57.907472 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:57.907364 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30"} err="failed to get container status \"e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30\": rpc error: code = NotFound desc = could not find container \"e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30\": container with ID starting with e2753d9c97505ccaf9c107b63689d2c73bae2fdda607b70de31e3ed3b4100b30 not found: ID does not exist" Oct 29 22:54:57.923942 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:54:57.923904 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9023-60de05\" is forbidden: User \"system:node:ip-10-0-134-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-94.ec2.internal' and this object" podUID="a0bca0c5-5002-4903-9408-db12765bf2fd" pod="opendatahub/test-curl-9023-60de05" Oct 29 22:54:58.839816 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:54:58.839783 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bca0c5-5002-4903-9408-db12765bf2fd" path="/var/lib/kubelet/pods/a0bca0c5-5002-4903-9408-db12765bf2fd/volumes" Oct 29 22:55:09.307783 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:55:09.307743 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:55:10.320129 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.320089 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm"] Oct 29 22:55:10.321103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.321077 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a0bca0c5-5002-4903-9408-db12765bf2fd" containerName="test-curl-9023-60de05" Oct 29 22:55:10.321103 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.321103 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bca0c5-5002-4903-9408-db12765bf2fd" containerName="test-curl-9023-60de05" Oct 29 22:55:10.321354 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.321340 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bca0c5-5002-4903-9408-db12765bf2fd" containerName="test-curl-9023-60de05" Oct 29 22:55:10.325431 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.325409 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.352761 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.352732 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm"] Oct 29 22:55:10.360899 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.360873 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-84304f-openshift-default-dockercfg-p57fc\"" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371546 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371657 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371743 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69w6q\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371778 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371818 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371850 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371921 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.371995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.373196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.372093 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.473925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.473887 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.473969 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69w6q\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474002 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474038 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474069 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474133 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474254 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474377 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474355 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474420 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474549 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474390 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.474968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.474937 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.475209 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.475022 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.475596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.475346 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.475596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.475591 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.477570 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.477546 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.477886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.477855 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.517176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.503604 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69w6q\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.517176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.510715 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:10.643046 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:10.642955 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:11.020834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:11.020751 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm"] Oct 29 22:55:11.043299 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:55:11.043258 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8862115c_5cd4_4636_bba6_16d04553feb9.slice/crio-8c6903f45e386d792a46896def128043fa4a3cefa03b9d62d5f04edb1f7e576a WatchSource:0}: Error finding container 8c6903f45e386d792a46896def128043fa4a3cefa03b9d62d5f04edb1f7e576a: Status 404 returned error can't find the container with id 8c6903f45e386d792a46896def128043fa4a3cefa03b9d62d5f04edb1f7e576a Oct 29 22:55:11.048295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:11.047455 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:55:11.048295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:11.047551 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:55:11.048295 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:11.047592 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:55:12.030188 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:12.030128 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" event={"ID":"8862115c-5cd4-4636-bba6-16d04553feb9","Type":"ContainerStarted","Data":"5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13"} Oct 29 22:55:12.030814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:12.030772 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" event={"ID":"8862115c-5cd4-4636-bba6-16d04553feb9","Type":"ContainerStarted","Data":"8c6903f45e386d792a46896def128043fa4a3cefa03b9d62d5f04edb1f7e576a"} Oct 29 22:55:12.093104 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:12.093043 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podStartSLOduration=2.093025441 podStartE2EDuration="2.093025441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:12.089982186 +0000 UTC m=+2427.405376273" watchObservedRunningTime="2026-10-29 22:55:12.093025441 +0000 UTC m=+2427.408419522" Oct 29 22:55:12.644098 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:12.644053 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:12.645901 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:12.645862 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:55:12.646039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:12.645932 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 22:55:13.643815 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:13.643772 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:55:13.644364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:13.643846 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 22:55:14.643798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:14.643761 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:55:14.644081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:14.643826 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 22:55:15.644251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:15.644217 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:55:15.644702 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:15.644289 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 22:55:16.643342 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:16.643306 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 22:55:16.643497 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:16.643361 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 22:55:17.962881 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:17.962850 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:17.963406 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:17.963376 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:17.964206 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:17.964184 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 22:55:22.122026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.121992 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw"] Oct 29 22:55:22.183280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.183252 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.183684 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.183663 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw"] Oct 29 22:55:22.208173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.206602 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7a5fe\"" Oct 29 22:55:22.208173 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.206772 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7a5fe-dockercfg-zf8tb\"" Oct 29 22:55:22.279461 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.279426 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.279631 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.279555 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstmt\" (UniqueName: \"kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.380938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.380866 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.381111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.380944 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstmt\" (UniqueName: \"kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.381708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.381682 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.401730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.401710 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstmt\" (UniqueName: \"kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.528897 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.528869 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:22.833969 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:55:22.833923 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf69404_feaf_4ed0_aaa7_59e784b2322e.slice/crio-f18fc583ada456289daf834bdea0a3139bc332f16c38fe13e0ee6278a40410b9 WatchSource:0}: Error finding container f18fc583ada456289daf834bdea0a3139bc332f16c38fe13e0ee6278a40410b9: Status 404 returned error can't find the container with id f18fc583ada456289daf834bdea0a3139bc332f16c38fe13e0ee6278a40410b9 Oct 29 22:55:22.843533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:22.843504 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw"] Oct 29 22:55:23.161544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:23.161469 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" event={"ID":"4cf69404-feaf-4ed0-aaa7-59e784b2322e","Type":"ContainerStarted","Data":"f18fc583ada456289daf834bdea0a3139bc332f16c38fe13e0ee6278a40410b9"} Oct 29 22:55:25.180465 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:25.180425 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" event={"ID":"4cf69404-feaf-4ed0-aaa7-59e784b2322e","Type":"ContainerStarted","Data":"95ff77b166de4347aad63b9684e8e250e265c45f1f619c9b0066bbc10f959157"} Oct 29 22:55:25.180866 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:25.180491 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:25.240802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:25.240754 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" podStartSLOduration=1.9287573390000001 podStartE2EDuration="3.240740431s" totalImagesPullingTime="1.311983092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:25.236683797 +0000 UTC m=+2440.552077878" watchObservedRunningTime="2026-10-29 22:55:25.240740431 +0000 UTC m=+2440.556134510" Oct 29 22:55:33.960267 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.960228 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt"] Oct 29 22:55:33.965957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.965929 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:33.979230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.979203 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt"] Oct 29 22:55:33.987076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.987048 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"openshift-service-ca.crt\"" Oct 29 22:55:33.987225 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.987102 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"kube-root-ca.crt\"" Oct 29 22:55:33.995798 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.995779 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"default-dockercfg-nn5ll\"" Oct 29 22:55:33.995913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:33.995842 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-dbb507\"/\"rate-test-model-dbb507-kserve-self-signed-certs\"" Oct 29 22:55:34.058459 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.058429 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.058569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.058510 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.058625 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.058609 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.058719 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.058701 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2mvx\" (UniqueName: \"kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.058757 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.058740 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.058910 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.058869 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.159888 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.159849 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.159950 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2mvx\" (UniqueName: \"kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.159997 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160081 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.160053 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.160276 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.160329 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.160347 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.160401 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.160938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.160672 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.162650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.162621 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.162962 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.162941 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.184527 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.184499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2mvx\" (UniqueName: \"kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx\") pod \"rate-test-model-dbb507-kserve-78685ddc6f-9gbxt\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.292236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.289790 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:34.622478 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:34.622447 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt"] Oct 29 22:55:34.623349 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:55:34.623316 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fe51827_6a65_4776_b12d_e9bd256b6888.slice/crio-ed04bd5531a5404e6ce0b1688fa7f44213c99f2bc9dbd72a5b97b814e8aafc5c WatchSource:0}: Error finding container ed04bd5531a5404e6ce0b1688fa7f44213c99f2bc9dbd72a5b97b814e8aafc5c: Status 404 returned error can't find the container with id ed04bd5531a5404e6ce0b1688fa7f44213c99f2bc9dbd72a5b97b814e8aafc5c Oct 29 22:55:35.272652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:35.272607 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" event={"ID":"0fe51827-6a65-4776-b12d-e9bd256b6888","Type":"ContainerStarted","Data":"a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7"} Oct 29 22:55:35.272652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:35.272658 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" event={"ID":"0fe51827-6a65-4776-b12d-e9bd256b6888","Type":"ContainerStarted","Data":"ed04bd5531a5404e6ce0b1688fa7f44213c99f2bc9dbd72a5b97b814e8aafc5c"} Oct 29 22:55:35.291125 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:35.291095 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:35.318708 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:35.318654 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" podStartSLOduration=2.318640625 podStartE2EDuration="2.318640625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:35.31543181 +0000 UTC m=+2450.630825894" watchObservedRunningTime="2026-10-29 22:55:35.318640625 +0000 UTC m=+2450.634034704" Oct 29 22:55:36.199902 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:36.199873 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:55:36.289731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:36.289695 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z"] Oct 29 22:55:37.095190 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:37.091546 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:55:39.472491 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:55:39.468506 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:55:44.290738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:44.290698 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:44.303731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:44.303708 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:44.442977 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:44.442944 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:55:48.051276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:48.051237 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft"] Oct 29 22:55:51.328297 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.328242 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" podUID="7d776659-9fd8-4dde-854d-5743feca1069" containerName="main" containerID="cri-o://ca5ac928ff0bbfb25de7ff924a8148f7a0faff44c444da59a963f33561e4e1f4" gracePeriod=45 Oct 29 22:55:51.493500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.493464 5073 generic.go:356] "Generic (PLEG): container finished" podID="7d776659-9fd8-4dde-854d-5743feca1069" containerID="ca5ac928ff0bbfb25de7ff924a8148f7a0faff44c444da59a963f33561e4e1f4" exitCode=0 Oct 29 22:55:51.493645 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.493518 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" event={"ID":"7d776659-9fd8-4dde-854d-5743feca1069","Type":"ContainerDied","Data":"ca5ac928ff0bbfb25de7ff924a8148f7a0faff44c444da59a963f33561e4e1f4"} Oct 29 22:55:51.611182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.611128 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:55:51.678500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678462 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5\" (UniqueName: \"kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5\") pod \"7d776659-9fd8-4dde-854d-5743feca1069\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " Oct 29 22:55:51.678663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678562 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm\") pod \"7d776659-9fd8-4dde-854d-5743feca1069\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " Oct 29 22:55:51.678663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678609 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache\") pod \"7d776659-9fd8-4dde-854d-5743feca1069\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " Oct 29 22:55:51.678795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678699 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir\") pod \"7d776659-9fd8-4dde-854d-5743feca1069\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " Oct 29 22:55:51.678855 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678826 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home\") pod \"7d776659-9fd8-4dde-854d-5743feca1069\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " Oct 29 22:55:51.678940 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678913 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache" pod "7d776659-9fd8-4dde-854d-5743feca1069" (UID: "7d776659-9fd8-4dde-854d-5743feca1069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:51.679016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.678942 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs\") pod \"7d776659-9fd8-4dde-854d-5743feca1069\" (UID: \"7d776659-9fd8-4dde-854d-5743feca1069\") " Oct 29 22:55:51.679069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.679024 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir" pod "7d776659-9fd8-4dde-854d-5743feca1069" (UID: "7d776659-9fd8-4dde-854d-5743feca1069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:51.679182 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.679128 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home" pod "7d776659-9fd8-4dde-854d-5743feca1069" (UID: "7d776659-9fd8-4dde-854d-5743feca1069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:51.679839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.679758 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:55:51.679839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.679785 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:55:51.679839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.679798 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:55:51.681424 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.681393 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5" pod "7d776659-9fd8-4dde-854d-5743feca1069" (UID: "7d776659-9fd8-4dde-854d-5743feca1069"). InnerVolumeSpecName "kube-api-access-4php5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:51.682085 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.682057 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm" pod "7d776659-9fd8-4dde-854d-5743feca1069" (UID: "7d776659-9fd8-4dde-854d-5743feca1069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:51.683365 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.683338 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs" pod "7d776659-9fd8-4dde-854d-5743feca1069" (UID: "7d776659-9fd8-4dde-854d-5743feca1069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:51.780733 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.780707 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4php5\" (UniqueName: \"kubernetes.io/projected/7d776659-9fd8-4dde-854d-5743feca1069-kube-api-access-4php5\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:55:51.780733 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.780734 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d776659-9fd8-4dde-854d-5743feca1069-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:55:51.780918 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:51.780744 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d776659-9fd8-4dde-854d-5743feca1069-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:55:52.502118 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.502089 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" Oct 29 22:55:52.502623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.502137 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z" event={"ID":"7d776659-9fd8-4dde-854d-5743feca1069","Type":"ContainerDied","Data":"57ba8350a5b71fa029248fa4cec084a9004770514cb10fa596dbf7215d2a24a9"} Oct 29 22:55:52.502623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.502216 5073 scope.go:122] "RemoveContainer" containerID="ca5ac928ff0bbfb25de7ff924a8148f7a0faff44c444da59a963f33561e4e1f4" Oct 29 22:55:52.563723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.563693 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z"] Oct 29 22:55:52.577973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.577940 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-afdbe8/rate-test-model-afdbe8-kserve-6b54547b9-2sr9z"] Oct 29 22:55:52.612369 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.612335 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk"] Oct 29 22:55:52.848826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:52.848738 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d776659-9fd8-4dde-854d-5743feca1069" path="/var/lib/kubelet/pods/7d776659-9fd8-4dde-854d-5743feca1069/volumes" Oct 29 22:55:56.238811 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.238724 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7"] Oct 29 22:55:56.239489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.239470 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7d776659-9fd8-4dde-854d-5743feca1069" containerName="main" Oct 29 22:55:56.239489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.239490 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d776659-9fd8-4dde-854d-5743feca1069" containerName="main" Oct 29 22:55:56.239654 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.239634 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d776659-9fd8-4dde-854d-5743feca1069" containerName="main" Oct 29 22:55:56.246041 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.246017 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.268164 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.268128 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7"] Oct 29 22:55:56.283108 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.283075 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2ae845-serving-cert\"" Oct 29 22:55:56.363076 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.363036 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpgqz\" (UniqueName: \"kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.363320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.363297 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.363401 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.363367 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.363562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.363535 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.465787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.465712 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpgqz\" (UniqueName: \"kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.466592 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.466553 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.466861 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.466830 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.467174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.467137 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.469204 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.469181 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.478413 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.478383 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.483582 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.483553 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.491606 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.491476 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpgqz\" (UniqueName: \"kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz\") pod \"maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.561255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.561226 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:56.864365 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:56.864326 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7"] Oct 29 22:55:56.864525 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:55:56.864503 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0fa98a_d57b_414d_b37b_1534ac637df5.slice/crio-fb374c3d1b7b28920fa9c439c1ef49af1457113e262a6381b93b0f6c751c3776 WatchSource:0}: Error finding container fb374c3d1b7b28920fa9c439c1ef49af1457113e262a6381b93b0f6c751c3776: Status 404 returned error can't find the container with id fb374c3d1b7b28920fa9c439c1ef49af1457113e262a6381b93b0f6c751c3776 Oct 29 22:55:57.619069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:57.618936 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" event={"ID":"7c0fa98a-d57b-414d-b37b-1534ac637df5","Type":"ContainerStarted","Data":"38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614"} Oct 29 22:55:57.619069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:57.619006 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" event={"ID":"7c0fa98a-d57b-414d-b37b-1534ac637df5","Type":"ContainerStarted","Data":"fb374c3d1b7b28920fa9c439c1ef49af1457113e262a6381b93b0f6c751c3776"} Oct 29 22:55:57.620127 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:57.620105 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:55:57.666054 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:55:57.665988 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" podStartSLOduration=1.321478631 podStartE2EDuration="1.665958369s" totalImagesPullingTime="344.479738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.661666408 +0000 UTC m=+2472.977060486" watchObservedRunningTime="2026-10-29 22:55:57.665958369 +0000 UTC m=+2472.981352453" Oct 29 22:56:03.081000 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.080946 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" podUID="2102a70f-da8f-4719-8ff4-d3481b9654fd" containerName="main" containerID="cri-o://c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21" gracePeriod=45 Oct 29 22:56:03.360839 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.360813 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:56:03.496307 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496276 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache\") pod \"2102a70f-da8f-4719-8ff4-d3481b9654fd\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " Oct 29 22:56:03.496537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496358 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm\") pod \"2102a70f-da8f-4719-8ff4-d3481b9654fd\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " Oct 29 22:56:03.496537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496379 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home\") pod \"2102a70f-da8f-4719-8ff4-d3481b9654fd\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " Oct 29 22:56:03.496537 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496428 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir\") pod \"2102a70f-da8f-4719-8ff4-d3481b9654fd\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " Oct 29 22:56:03.496773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496577 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs\") pod \"2102a70f-da8f-4719-8ff4-d3481b9654fd\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " Oct 29 22:56:03.496773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496614 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache" pod "2102a70f-da8f-4719-8ff4-d3481b9654fd" (UID: "2102a70f-da8f-4719-8ff4-d3481b9654fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:03.496773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496662 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n\" (UniqueName: \"kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n\") pod \"2102a70f-da8f-4719-8ff4-d3481b9654fd\" (UID: \"2102a70f-da8f-4719-8ff4-d3481b9654fd\") " Oct 29 22:56:03.496773 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496643 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home" pod "2102a70f-da8f-4719-8ff4-d3481b9654fd" (UID: "2102a70f-da8f-4719-8ff4-d3481b9654fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:03.496952 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.496732 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir" pod "2102a70f-da8f-4719-8ff4-d3481b9654fd" (UID: "2102a70f-da8f-4719-8ff4-d3481b9654fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:03.497548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.497521 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:03.497548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.497549 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:03.497548 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.497562 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:03.498789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.498766 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs" pod "2102a70f-da8f-4719-8ff4-d3481b9654fd" (UID: "2102a70f-da8f-4719-8ff4-d3481b9654fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:03.498789 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.498777 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm" pod "2102a70f-da8f-4719-8ff4-d3481b9654fd" (UID: "2102a70f-da8f-4719-8ff4-d3481b9654fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:03.498937 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.498915 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n" pod "2102a70f-da8f-4719-8ff4-d3481b9654fd" (UID: "2102a70f-da8f-4719-8ff4-d3481b9654fd"). InnerVolumeSpecName "kube-api-access-4xd6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:03.598681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.598613 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2102a70f-da8f-4719-8ff4-d3481b9654fd-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:03.598681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.598640 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2102a70f-da8f-4719-8ff4-d3481b9654fd-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:03.598681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.598655 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xd6n\" (UniqueName: \"kubernetes.io/projected/2102a70f-da8f-4719-8ff4-d3481b9654fd-kube-api-access-4xd6n\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:03.715584 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.715561 5073 generic.go:356] "Generic (PLEG): container finished" podID="2102a70f-da8f-4719-8ff4-d3481b9654fd" containerID="c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21" exitCode=0 Oct 29 22:56:03.715678 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.715602 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" Oct 29 22:56:03.715678 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.715655 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" event={"ID":"2102a70f-da8f-4719-8ff4-d3481b9654fd","Type":"ContainerDied","Data":"c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21"} Oct 29 22:56:03.715758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.715695 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft" event={"ID":"2102a70f-da8f-4719-8ff4-d3481b9654fd","Type":"ContainerDied","Data":"36c2b67e3f234169589606b94952cf05f57f7dccadba2445ae6c24db9f894b87"} Oct 29 22:56:03.715758 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.715722 5073 scope.go:122] "RemoveContainer" containerID="c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21" Oct 29 22:56:03.726663 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.726640 5073 scope.go:122] "RemoveContainer" containerID="c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21" Oct 29 22:56:03.726948 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:56:03.726927 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21\": container with ID starting with c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21 not found: ID does not exist" containerID="c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21" Oct 29 22:56:03.727033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.726960 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21"} err="failed to get container status \"c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21\": rpc error: code = NotFound desc = could not find container \"c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21\": container with ID starting with c0e1c0c3bc6baeeb7ca66f265ca515630d8b6ed1ce238f781c6292f1c0f18f21 not found: ID does not exist" Oct 29 22:56:03.776345 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.776320 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft"] Oct 29 22:56:03.782099 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:03.782079 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-84304f/rate-test-model-84304f-kserve-665c74ff49-8wmft"] Oct 29 22:56:04.641276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:04.641239 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 22:56:04.839627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:04.839598 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2102a70f-da8f-4719-8ff4-d3481b9654fd" path="/var/lib/kubelet/pods/2102a70f-da8f-4719-8ff4-d3481b9654fd/volumes" Oct 29 22:56:07.646973 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:07.639582 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" podUID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" containerName="main" containerID="cri-o://f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221" gracePeriod=45 Oct 29 22:56:07.764961 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:07.764928 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" podUID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.147:8000/ready\": dial tcp 10.132.0.147:8000: connect: connection refused" Oct 29 22:56:07.967302 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:07.967272 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:56:08.034968 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.034934 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm\") pod \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " Oct 29 22:56:08.035167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035007 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache\") pod \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " Oct 29 22:56:08.035167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035045 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs\") pod \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " Oct 29 22:56:08.035167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035101 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r\" (UniqueName: \"kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r\") pod \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " Oct 29 22:56:08.035356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035195 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir\") pod \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " Oct 29 22:56:08.035356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035257 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home\") pod \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\" (UID: \"b67510dd-20c1-4c8b-818c-5c7ce6d2022d\") " Oct 29 22:56:08.035752 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035721 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache" pod "b67510dd-20c1-4c8b-818c-5c7ce6d2022d" (UID: "b67510dd-20c1-4c8b-818c-5c7ce6d2022d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:08.035948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035926 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home" pod "b67510dd-20c1-4c8b-818c-5c7ce6d2022d" (UID: "b67510dd-20c1-4c8b-818c-5c7ce6d2022d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:08.036027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.035939 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir" pod "b67510dd-20c1-4c8b-818c-5c7ce6d2022d" (UID: "b67510dd-20c1-4c8b-818c-5c7ce6d2022d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:08.036134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.036089 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:08.036134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.036109 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:08.036134 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.036122 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:08.038087 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.038056 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs" pod "b67510dd-20c1-4c8b-818c-5c7ce6d2022d" (UID: "b67510dd-20c1-4c8b-818c-5c7ce6d2022d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:08.038333 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.038310 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm" pod "b67510dd-20c1-4c8b-818c-5c7ce6d2022d" (UID: "b67510dd-20c1-4c8b-818c-5c7ce6d2022d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:08.038418 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.038307 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r" pod "b67510dd-20c1-4c8b-818c-5c7ce6d2022d" (UID: "b67510dd-20c1-4c8b-818c-5c7ce6d2022d"). InnerVolumeSpecName "kube-api-access-fvg6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:08.137719 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.137683 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:08.137719 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.137712 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvg6r\" (UniqueName: \"kubernetes.io/projected/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-kube-api-access-fvg6r\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:08.137719 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.137722 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67510dd-20c1-4c8b-818c-5c7ce6d2022d-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:56:08.764201 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.764164 5073 generic.go:356] "Generic (PLEG): container finished" podID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" containerID="f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221" exitCode=0 Oct 29 22:56:08.764694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.764236 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" Oct 29 22:56:08.764694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.764243 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" event={"ID":"b67510dd-20c1-4c8b-818c-5c7ce6d2022d","Type":"ContainerDied","Data":"f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221"} Oct 29 22:56:08.764694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.764303 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk" event={"ID":"b67510dd-20c1-4c8b-818c-5c7ce6d2022d","Type":"ContainerDied","Data":"18da9439075ee82e95f235cd6ff5993963fc2f8f03330f2db35c4a3be46c7cb1"} Oct 29 22:56:08.764694 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.764328 5073 scope.go:122] "RemoveContainer" containerID="f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221" Oct 29 22:56:08.775882 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.775866 5073 scope.go:122] "RemoveContainer" containerID="f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221" Oct 29 22:56:08.776186 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:56:08.776137 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221\": container with ID starting with f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221 not found: ID does not exist" containerID="f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221" Oct 29 22:56:08.776255 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.776195 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221"} err="failed to get container status \"f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221\": rpc error: code = NotFound desc = could not find container \"f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221\": container with ID starting with f6ef76d382cb3edcb2572f77ead8e7af41f9c611f05ba5720c994eabe77ae221 not found: ID does not exist" Oct 29 22:56:08.821802 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.821769 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk"] Oct 29 22:56:08.839228 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:08.839207 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-e81b56/sub-test-model-e81b56-kserve-85cbfbc59f-nvgbk"] Oct 29 22:56:09.256418 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:56:09.256368 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:56:10.839213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:10.839182 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" path="/var/lib/kubelet/pods/b67510dd-20c1-4c8b-818c-5c7ce6d2022d/volumes" Oct 29 22:56:35.531925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.531888 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp"] Oct 29 22:56:35.532574 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.532557 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2102a70f-da8f-4719-8ff4-d3481b9654fd" containerName="main" Oct 29 22:56:35.532635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.532575 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2102a70f-da8f-4719-8ff4-d3481b9654fd" containerName="main" Oct 29 22:56:35.532635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.532627 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" containerName="main" Oct 29 22:56:35.532635 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.532633 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" containerName="main" Oct 29 22:56:35.532754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.532743 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2102a70f-da8f-4719-8ff4-d3481b9654fd" containerName="main" Oct 29 22:56:35.532812 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.532766 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67510dd-20c1-4c8b-818c-5c7ce6d2022d" containerName="main" Oct 29 22:56:35.536713 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.536694 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.555033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.555008 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp"] Oct 29 22:56:35.556492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.556473 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c6f83c-serving-cert\"" Oct 29 22:56:35.656614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.656587 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.656763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.656654 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsp8l\" (UniqueName: \"kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.656763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.656739 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.656912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.656896 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.758834 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.758771 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.759492 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.759454 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsp8l\" (UniqueName: \"kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.760181 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.760161 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.760545 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.760527 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.761996 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.761640 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.764830 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.763637 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.766956 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.766912 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.781260 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.781235 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsp8l\" (UniqueName: \"kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l\") pod \"maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:35.850396 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:35.850323 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:36.153082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:36.153056 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp"] Oct 29 22:56:36.154711 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:56:36.154682 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0891be57_bb2a_4114_9852_bfb2b82c7a31.slice/crio-6012e813fb460fabf2fc4dbd8527b375ea64c66937c62e8df59bb2819d6aadfd WatchSource:0}: Error finding container 6012e813fb460fabf2fc4dbd8527b375ea64c66937c62e8df59bb2819d6aadfd: Status 404 returned error can't find the container with id 6012e813fb460fabf2fc4dbd8527b375ea64c66937c62e8df59bb2819d6aadfd Oct 29 22:56:37.030122 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:37.030082 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" event={"ID":"0891be57-bb2a-4114-9852-bfb2b82c7a31","Type":"ContainerStarted","Data":"ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9"} Oct 29 22:56:37.030581 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:37.030219 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:37.030581 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:37.030240 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" event={"ID":"0891be57-bb2a-4114-9852-bfb2b82c7a31","Type":"ContainerStarted","Data":"6012e813fb460fabf2fc4dbd8527b375ea64c66937c62e8df59bb2819d6aadfd"} Oct 29 22:56:37.072557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:37.072512 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" podStartSLOduration=1.6898395160000002 podStartE2EDuration="2.07250142s" totalImagesPullingTime="382.661904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:37.071722519 +0000 UTC m=+2512.387116597" watchObservedRunningTime="2026-10-29 22:56:37.07250142 +0000 UTC m=+2512.387895494" Oct 29 22:56:39.256687 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:56:39.256650 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:56:43.043669 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:43.043640 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 22:56:48.010189 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.010133 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp"] Oct 29 22:56:48.015870 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.015852 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.035904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.035871 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp"] Oct 29 22:56:48.043434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.043409 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a15f4\"/\"kube-root-ca.crt\"" Oct 29 22:56:48.043576 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.043456 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a15f4\"/\"default-dockercfg-kpkl5\"" Oct 29 22:56:48.048035 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.048010 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a15f4\"/\"openshift-service-ca.crt\"" Oct 29 22:56:48.048220 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.048059 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a15f4\"/\"auth-test-model-9a15f4-kserve-self-signed-certs\"" Oct 29 22:56:48.077242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.077215 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.077361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.077344 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.077422 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.077371 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wg9z\" (UniqueName: \"kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.077476 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.077459 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.077544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.077532 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.077595 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.077583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.178827 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.178794 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.178985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.178891 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.178985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.178971 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.179121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.178994 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wg9z\" (UniqueName: \"kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.179121 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.179067 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.179259 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.179123 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.179450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.179412 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.179920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.179893 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.180095 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.179970 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.181683 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.181659 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.182339 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.182224 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.200829 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.200803 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wg9z\" (UniqueName: \"kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z\") pod \"auth-test-model-9a15f4-kserve-654f88d7cc-dthbp\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.342550 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.342450 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:48.716511 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:48.716484 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp"] Oct 29 22:56:49.132670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:49.132630 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" event={"ID":"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1","Type":"ContainerStarted","Data":"44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e"} Oct 29 22:56:49.133202 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:49.132682 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" event={"ID":"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1","Type":"ContainerStarted","Data":"e8a24d905f6b1de0bd1e3199f40903a6cfcbfb7d4b05e45c35743175b3513710"} Oct 29 22:56:49.188123 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:49.188074 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" podStartSLOduration=2.188059103 podStartE2EDuration="2.188059103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:49.183384907 +0000 UTC m=+2524.498779004" watchObservedRunningTime="2026-10-29 22:56:49.188059103 +0000 UTC m=+2524.503453183" Oct 29 22:56:49.343853 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:49.343787 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:58.342909 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:58.342868 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:58.355993 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:58.355966 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:56:59.019990 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:59.019950 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:56:59.221131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:56:59.221095 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 22:57:09.255902 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:57:09.255861 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:57:17.190596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.190539 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq"] Oct 29 22:57:17.195920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.195896 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.218193 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.216222 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq"] Oct 29 22:57:17.225723 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.225688 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-36b46d\"/\"openshift-service-ca.crt\"" Oct 29 22:57:17.225897 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.225769 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-36b46d\"/\"kube-root-ca.crt\"" Oct 29 22:57:17.229039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.229016 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-36b46d\"/\"auth-test-model-36b46d-kserve-self-signed-certs\"" Oct 29 22:57:17.229167 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.229092 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-36b46d\"/\"default-dockercfg-q9cm7\"" Oct 29 22:57:17.323280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.323234 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.323280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.323277 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.323533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.323424 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.323626 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.323609 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.323685 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.323645 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.323742 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.323681 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzxj\" (UniqueName: \"kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.425524 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425487 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.425717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425533 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.425717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425559 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzxj\" (UniqueName: \"kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.425717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425633 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.425717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425649 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.425963 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425723 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.426034 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.425952 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.426097 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.426057 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.426208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.426186 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.428480 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.428450 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.428667 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.428642 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.452246 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.452167 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzxj\" (UniqueName: \"kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj\") pod \"auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.525777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.525450 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:17.879974 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:17.879859 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq"] Oct 29 22:57:17.880998 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:57:17.880970 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aaa2d63_47c8_49bd_a9ba_ac1dbdd84b83.slice/crio-320d34acf7d63871e9bc40c5b80532ed577a7f0d432cb45b20306aeee60ade5f WatchSource:0}: Error finding container 320d34acf7d63871e9bc40c5b80532ed577a7f0d432cb45b20306aeee60ade5f: Status 404 returned error can't find the container with id 320d34acf7d63871e9bc40c5b80532ed577a7f0d432cb45b20306aeee60ade5f Oct 29 22:57:18.358854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:18.358817 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" event={"ID":"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83","Type":"ContainerStarted","Data":"b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f"} Oct 29 22:57:18.359388 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:18.358865 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" event={"ID":"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83","Type":"ContainerStarted","Data":"320d34acf7d63871e9bc40c5b80532ed577a7f0d432cb45b20306aeee60ade5f"} Oct 29 22:57:18.404848 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:18.404793 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" podStartSLOduration=1.404775964 podStartE2EDuration="1.404775964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:18.400751631 +0000 UTC m=+2553.716145712" watchObservedRunningTime="2026-10-29 22:57:18.404775964 +0000 UTC m=+2553.720170045" Oct 29 22:57:18.527168 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:18.527114 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:21.082741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.082710 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb"] Oct 29 22:57:21.087729 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.087703 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.118484 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.118462 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-329f7b\"/\"openshift-service-ca.crt\"" Oct 29 22:57:21.130011 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.129991 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-329f7b\"/\"kube-root-ca.crt\"" Oct 29 22:57:21.130362 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.130338 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-329f7b\"/\"auth-test-model-329f7b-kserve-self-signed-certs\"" Oct 29 22:57:21.132131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.132109 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-329f7b\"/\"default-dockercfg-dl2gk\"" Oct 29 22:57:21.135985 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.135960 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb"] Oct 29 22:57:21.179020 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.178983 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.179272 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.179250 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.179364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.179283 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.179364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.179314 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.179483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.179401 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.179483 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.179444 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gw84\" (UniqueName: \"kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.280814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.280783 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.280930 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.280947 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.280967 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281003 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.280993 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281283 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.281033 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gw84\" (UniqueName: \"kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.281297 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.281343 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.281743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.281716 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.283555 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.283524 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.284263 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.284247 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.304530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.304501 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gw84\" (UniqueName: \"kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84\") pod \"auth-test-model-329f7b-kserve-54997dc6fc-m9cqb\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.410223 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.406960 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:21.810911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:21.810878 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb"] Oct 29 22:57:21.814236 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:57:21.814208 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20bbe73c_4a61_477c_a505_91bdf13d1dd3.slice/crio-c0e1413cbe640d9094a7bc06e031685c8b3ddca240137d7b6ab5c4d1bd20b9f9 WatchSource:0}: Error finding container c0e1413cbe640d9094a7bc06e031685c8b3ddca240137d7b6ab5c4d1bd20b9f9: Status 404 returned error can't find the container with id c0e1413cbe640d9094a7bc06e031685c8b3ddca240137d7b6ab5c4d1bd20b9f9 Oct 29 22:57:22.415897 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:22.415863 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" event={"ID":"20bbe73c-4a61-477c-a505-91bdf13d1dd3","Type":"ContainerStarted","Data":"7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e"} Oct 29 22:57:22.416434 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:22.415906 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" event={"ID":"20bbe73c-4a61-477c-a505-91bdf13d1dd3","Type":"ContainerStarted","Data":"c0e1413cbe640d9094a7bc06e031685c8b3ddca240137d7b6ab5c4d1bd20b9f9"} Oct 29 22:57:22.466436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:22.466383 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" podStartSLOduration=1.466367848 podStartE2EDuration="1.466367848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:22.460629334 +0000 UTC m=+2557.776023413" watchObservedRunningTime="2026-10-29 22:57:22.466367848 +0000 UTC m=+2557.781761947" Oct 29 22:57:23.407826 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:23.407788 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:27.527623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:27.527536 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:27.540879 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:27.540850 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:28.489938 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:28.489908 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 22:57:31.407651 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:31.407616 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:31.420544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:31.420513 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:31.567069 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:31.567039 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 22:57:32.271814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.271781 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct"] Oct 29 22:57:32.277748 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.277721 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.298557 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.298531 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct"] Oct 29 22:57:32.308387 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.308360 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2ae845\"/\"kube-root-ca.crt\"" Oct 29 22:57:32.316738 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.316695 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2ae845\"/\"default-dockercfg-tmqsz\"" Oct 29 22:57:32.319356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.319322 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2ae845\"/\"openshift-service-ca.crt\"" Oct 29 22:57:32.319665 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.319602 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2ae845\"/\"auth-test-model-2ae845-kserve-self-signed-certs\"" Oct 29 22:57:32.373040 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.372995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.373303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.373050 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.373303 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.373204 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4np\" (UniqueName: \"kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.373610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.373356 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.373610 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.373489 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.373730 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.373710 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.474652 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.474615 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.474746 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.474882 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.474940 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.474972 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.475021 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4np\" (UniqueName: \"kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.475075 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.475084 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.475782 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.475750 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.477988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.477958 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.478186 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.478126 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.497131 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.497099 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4np\" (UniqueName: \"kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np\") pod \"auth-test-model-2ae845-kserve-5c4cc4d689-cwlct\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:32.597021 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:32.595552 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:33.062983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:33.062955 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct"] Oct 29 22:57:33.064239 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:57:33.064207 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod780ac08b_7550_49ee_8b62_141339eb9ff7.slice/crio-6cbb1369d32aadeed18dacddc7dec3d2949795f0763928f60a27d0e62242c372 WatchSource:0}: Error finding container 6cbb1369d32aadeed18dacddc7dec3d2949795f0763928f60a27d0e62242c372: Status 404 returned error can't find the container with id 6cbb1369d32aadeed18dacddc7dec3d2949795f0763928f60a27d0e62242c372 Oct 29 22:57:33.608670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:33.608632 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" event={"ID":"780ac08b-7550-49ee-8b62-141339eb9ff7","Type":"ContainerStarted","Data":"f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b"} Oct 29 22:57:33.608670 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:33.608676 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" event={"ID":"780ac08b-7550-49ee-8b62-141339eb9ff7","Type":"ContainerStarted","Data":"6cbb1369d32aadeed18dacddc7dec3d2949795f0763928f60a27d0e62242c372"} Oct 29 22:57:33.654775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:33.654727 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" podStartSLOduration=1.654713337 podStartE2EDuration="1.654713337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:33.650807907 +0000 UTC m=+2568.966201985" watchObservedRunningTime="2026-10-29 22:57:33.654713337 +0000 UTC m=+2568.970107451" Oct 29 22:57:34.599771 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:34.599734 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:37.427704 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:37.427294 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw"] Oct 29 22:57:37.428258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:37.427917 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" podUID="4cf69404-feaf-4ed0-aaa7-59e784b2322e" containerName="payload-pre-processing" containerID="cri-o://95ff77b166de4347aad63b9684e8e250e265c45f1f619c9b0066bbc10f959157" gracePeriod=30 Oct 29 22:57:37.656588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:37.644868 5073 generic.go:356] "Generic (PLEG): container finished" podID="4cf69404-feaf-4ed0-aaa7-59e784b2322e" containerID="95ff77b166de4347aad63b9684e8e250e265c45f1f619c9b0066bbc10f959157" exitCode=0 Oct 29 22:57:37.656588 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:37.644966 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" event={"ID":"4cf69404-feaf-4ed0-aaa7-59e784b2322e","Type":"ContainerDied","Data":"95ff77b166de4347aad63b9684e8e250e265c45f1f619c9b0066bbc10f959157"} Oct 29 22:57:37.941913 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:37.941248 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:57:38.033818 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.033781 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume\") pod \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " Oct 29 22:57:38.034033 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.033859 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt\" (UniqueName: \"kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt\") pod \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\" (UID: \"4cf69404-feaf-4ed0-aaa7-59e784b2322e\") " Oct 29 22:57:38.034311 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.034280 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume" pod "4cf69404-feaf-4ed0-aaa7-59e784b2322e" (UID: "4cf69404-feaf-4ed0-aaa7-59e784b2322e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:38.034613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.034592 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf69404-feaf-4ed0-aaa7-59e784b2322e-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:57:38.036450 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.036422 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt" pod "4cf69404-feaf-4ed0-aaa7-59e784b2322e" (UID: "4cf69404-feaf-4ed0-aaa7-59e784b2322e"). InnerVolumeSpecName "kube-api-access-sstmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:38.136530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.136464 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstmt\" (UniqueName: \"kubernetes.io/projected/4cf69404-feaf-4ed0-aaa7-59e784b2322e-kube-api-access-sstmt\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:57:38.653591 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.653560 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" Oct 29 22:57:38.654094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.653560 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw" event={"ID":"4cf69404-feaf-4ed0-aaa7-59e784b2322e","Type":"ContainerDied","Data":"f18fc583ada456289daf834bdea0a3139bc332f16c38fe13e0ee6278a40410b9"} Oct 29 22:57:38.654094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.653687 5073 scope.go:122] "RemoveContainer" containerID="95ff77b166de4347aad63b9684e8e250e265c45f1f619c9b0066bbc10f959157" Oct 29 22:57:38.738045 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.738021 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw"] Oct 29 22:57:38.752485 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.752461 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5f858d98c-zkgfw"] Oct 29 22:57:38.840596 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:38.840570 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf69404-feaf-4ed0-aaa7-59e784b2322e" path="/var/lib/kubelet/pods/4cf69404-feaf-4ed0-aaa7-59e784b2322e/volumes" Oct 29 22:57:39.257385 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:57:39.257348 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:57:42.599833 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:42.599796 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:42.612699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:42.612673 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:42.692028 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:42.692000 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 22:57:48.371894 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:57:48.371853 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:58:09.366522 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:58:09.366478 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:58:11.661012 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.660977 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2"] Oct 29 22:58:11.662299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.662273 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4cf69404-feaf-4ed0-aaa7-59e784b2322e" containerName="payload-pre-processing" Oct 29 22:58:11.662299 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.662297 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf69404-feaf-4ed0-aaa7-59e784b2322e" containerName="payload-pre-processing" Oct 29 22:58:11.662523 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.662507 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf69404-feaf-4ed0-aaa7-59e784b2322e" containerName="payload-pre-processing" Oct 29 22:58:11.666657 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.666637 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.705904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.705876 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2"] Oct 29 22:58:11.714082 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.714055 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e803b6\"/\"openshift-service-ca.crt\"" Oct 29 22:58:11.714276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.714056 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e803b6\"/\"auth-test-model-e803b6-kserve-self-signed-certs\"" Oct 29 22:58:11.714276 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.714109 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e803b6\"/\"kube-root-ca.crt\"" Oct 29 22:58:11.714462 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.714073 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e803b6\"/\"default-dockercfg-jqs9w\"" Oct 29 22:58:11.762613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.762577 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzs4\" (UniqueName: \"kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.762787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.762714 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.762787 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.762772 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.762922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.762907 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.762976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.762965 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.763086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.763064 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.864482 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.864446 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.864659 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.864609 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.864734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.864690 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.864795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.864733 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.864859 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.864799 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzs4\" (UniqueName: \"kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.864935 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.864919 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.865265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.865239 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.865627 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.865389 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.865763 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.865711 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.869644 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.869619 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.870208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.870184 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:11.890120 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:11.890085 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzs4\" (UniqueName: \"kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4\") pod \"auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:12.018174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:12.014440 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:12.405951 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:12.405914 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2"] Oct 29 22:58:12.406109 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:58:12.406067 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20c8abc4_2229_4a9d_ab4d_f57b4a79a5ab.slice/crio-31e5c33beac23b5eed5a5a55f0fc942dade3b33ccdc7918f81b5e9fc6b8ac1a3 WatchSource:0}: Error finding container 31e5c33beac23b5eed5a5a55f0fc942dade3b33ccdc7918f81b5e9fc6b8ac1a3: Status 404 returned error can't find the container with id 31e5c33beac23b5eed5a5a55f0fc942dade3b33ccdc7918f81b5e9fc6b8ac1a3 Oct 29 22:58:12.940690 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:12.940657 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" event={"ID":"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab","Type":"ContainerStarted","Data":"eebd23514edd354826d3e704d80a093528f3e9f930c0de869ff756af8f469ae6"} Oct 29 22:58:12.941088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:12.940702 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" event={"ID":"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab","Type":"ContainerStarted","Data":"31e5c33beac23b5eed5a5a55f0fc942dade3b33ccdc7918f81b5e9fc6b8ac1a3"} Oct 29 22:58:13.001247 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:13.001199 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" podStartSLOduration=2.001184812 podStartE2EDuration="2.001184812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:12.999353468 +0000 UTC m=+2608.314747550" watchObservedRunningTime="2026-10-29 22:58:13.001184812 +0000 UTC m=+2608.316578893" Oct 29 22:58:13.015851 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:13.015826 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:22.015858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:22.015830 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:22.029513 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:22.029488 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:22.988325 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:22.987308 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz"] Oct 29 22:58:22.993517 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:22.993494 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.023292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.021271 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz"] Oct 29 22:58:23.029218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.028787 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1dabb8-dockercfg-m759l\"" Oct 29 22:58:23.029218 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.028853 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1dabb8\"" Oct 29 22:58:23.035606 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.035580 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 22:58:23.081742 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.081712 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.081904 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.081883 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gns5d\" (UniqueName: \"kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d\") pod \"payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.183338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.183301 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns5d\" (UniqueName: \"kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d\") pod \"payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.183593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.183574 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.184516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.184489 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.205252 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.205231 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gns5d\" (UniqueName: \"kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d\") pod \"payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.310290 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.310220 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:23.615249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:23.615224 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz"] Oct 29 22:58:23.615377 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:58:23.615290 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5c6ff7_ca44_45c6_bc98_c27a3a24f54e.slice/crio-b4ef498f3b19830c3799ad3f601057630a29aac9d506658a3b60dc931a04203d WatchSource:0}: Error finding container b4ef498f3b19830c3799ad3f601057630a29aac9d506658a3b60dc931a04203d: Status 404 returned error can't find the container with id b4ef498f3b19830c3799ad3f601057630a29aac9d506658a3b60dc931a04203d Oct 29 22:58:24.043851 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:24.043815 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" event={"ID":"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e","Type":"ContainerStarted","Data":"d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f"} Oct 29 22:58:24.044324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:24.043915 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:24.044324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:24.043940 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" event={"ID":"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e","Type":"ContainerStarted","Data":"b4ef498f3b19830c3799ad3f601057630a29aac9d506658a3b60dc931a04203d"} Oct 29 22:58:24.102907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:24.102853 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" podStartSLOduration=2.102839878 podStartE2EDuration="2.102839878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:24.099624691 +0000 UTC m=+2619.415018782" watchObservedRunningTime="2026-10-29 22:58:24.102839878 +0000 UTC m=+2619.418233958" Oct 29 22:58:32.229854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:32.218651 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj"] Oct 29 22:58:35.053982 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:35.053948 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 22:58:39.257346 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:58:39.257306 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:58:44.800446 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:44.800403 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:58:47.256274 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.256218 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" podUID="4ac4a115-a240-459c-9853-d9d11faa6df7" containerName="main" containerID="cri-o://c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23" gracePeriod=45 Oct 29 22:58:47.684174 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.677054 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:58:47.742965 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.742932 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir\") pod \"4ac4a115-a240-459c-9853-d9d11faa6df7\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " Oct 29 22:58:47.743230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.743046 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm\") pod \"4ac4a115-a240-459c-9853-d9d11faa6df7\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " Oct 29 22:58:47.743230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.743092 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache\") pod \"4ac4a115-a240-459c-9853-d9d11faa6df7\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " Oct 29 22:58:47.743230 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.743192 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8\" (UniqueName: \"kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8\") pod \"4ac4a115-a240-459c-9853-d9d11faa6df7\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " Oct 29 22:58:47.743436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.743278 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs\") pod \"4ac4a115-a240-459c-9853-d9d11faa6df7\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " Oct 29 22:58:47.743436 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.743304 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home\") pod \"4ac4a115-a240-459c-9853-d9d11faa6df7\" (UID: \"4ac4a115-a240-459c-9853-d9d11faa6df7\") " Oct 29 22:58:47.744301 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.744264 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home" pod "4ac4a115-a240-459c-9853-d9d11faa6df7" (UID: "4ac4a115-a240-459c-9853-d9d11faa6df7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.744717 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.744689 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache" pod "4ac4a115-a240-459c-9853-d9d11faa6df7" (UID: "4ac4a115-a240-459c-9853-d9d11faa6df7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.747634 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.747273 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8" pod "4ac4a115-a240-459c-9853-d9d11faa6df7" (UID: "4ac4a115-a240-459c-9853-d9d11faa6df7"). InnerVolumeSpecName "kube-api-access-8d8b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:47.747634 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.747537 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir" pod "4ac4a115-a240-459c-9853-d9d11faa6df7" (UID: "4ac4a115-a240-459c-9853-d9d11faa6df7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.748516 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.748474 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm" pod "4ac4a115-a240-459c-9853-d9d11faa6df7" (UID: "4ac4a115-a240-459c-9853-d9d11faa6df7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:58:47.748623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.748524 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs" pod "4ac4a115-a240-459c-9853-d9d11faa6df7" (UID: "4ac4a115-a240-459c-9853-d9d11faa6df7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:47.845539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.845471 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac4a115-a240-459c-9853-d9d11faa6df7-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.845539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.845501 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.845539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.845529 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.845539 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.845541 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.845749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.845552 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac4a115-a240-459c-9853-d9d11faa6df7-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.845749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:47.845565 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8b8\" (UniqueName: \"kubernetes.io/projected/4ac4a115-a240-459c-9853-d9d11faa6df7-kube-api-access-8d8b8\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:58:48.239601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.239563 5073 generic.go:356] "Generic (PLEG): container finished" podID="4ac4a115-a240-459c-9853-d9d11faa6df7" containerID="c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23" exitCode=0 Oct 29 22:58:48.239800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.239612 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" Oct 29 22:58:48.239800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.239657 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" event={"ID":"4ac4a115-a240-459c-9853-d9d11faa6df7","Type":"ContainerDied","Data":"c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23"} Oct 29 22:58:48.239800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.239712 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj" event={"ID":"4ac4a115-a240-459c-9853-d9d11faa6df7","Type":"ContainerDied","Data":"c2a18c8891ba3fcdd7b1e180c2628f0f02cf968df776d6ba696df567a985cf2c"} Oct 29 22:58:48.239800 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.239734 5073 scope.go:122] "RemoveContainer" containerID="c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23" Oct 29 22:58:48.251593 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.251569 5073 scope.go:122] "RemoveContainer" containerID="c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23" Oct 29 22:58:48.251909 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:58:48.251886 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23\": container with ID starting with c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23 not found: ID does not exist" containerID="c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23" Oct 29 22:58:48.251959 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.251921 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23"} err="failed to get container status \"c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23\": rpc error: code = NotFound desc = could not find container \"c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23\": container with ID starting with c0063a21878bafe1e2a6836f56e5b1e918946fffac4755d0bdc775ef0c5d4e23 not found: ID does not exist" Oct 29 22:58:48.299794 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.299768 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj"] Oct 29 22:58:48.303027 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.303008 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/rate-test-model-437981-kserve-857f5cfbfc-2lkgj"] Oct 29 22:58:48.839320 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:48.839291 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac4a115-a240-459c-9853-d9d11faa6df7" path="/var/lib/kubelet/pods/4ac4a115-a240-459c-9853-d9d11faa6df7/volumes" Oct 29 22:58:57.271840 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:58:57.271748 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt"] Oct 29 22:59:00.949737 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:00.949697 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh"] Oct 29 22:59:00.950289 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:00.949930 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" podUID="477ffde3-c6d7-4a62-9f90-6b6529e80df1" containerName="maas-api" containerID="cri-o://6fa68a2a0f2a12d9cc939db39cbbca33d8b1b2f059e60000f47352efd58c05d1" gracePeriod=30 Oct 29 22:59:01.553768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.553727 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:59:01.587741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.585790 5073 generic.go:356] "Generic (PLEG): container finished" podID="477ffde3-c6d7-4a62-9f90-6b6529e80df1" containerID="6fa68a2a0f2a12d9cc939db39cbbca33d8b1b2f059e60000f47352efd58c05d1" exitCode=0 Oct 29 22:59:01.587741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.585883 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" event={"ID":"477ffde3-c6d7-4a62-9f90-6b6529e80df1","Type":"ContainerDied","Data":"6fa68a2a0f2a12d9cc939db39cbbca33d8b1b2f059e60000f47352efd58c05d1"} Oct 29 22:59:01.587741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.585936 5073 scope.go:122] "RemoveContainer" containerID="6fa68a2a0f2a12d9cc939db39cbbca33d8b1b2f059e60000f47352efd58c05d1" Oct 29 22:59:01.587741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.586089 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh" Oct 29 22:59:01.591681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.589789 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls\") pod \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " Oct 29 22:59:01.591681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.589978 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89\" (UniqueName: \"kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89\") pod \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " Oct 29 22:59:01.591681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.590020 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs\") pod \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " Oct 29 22:59:01.591681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.590130 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca\") pod \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\" (UID: \"477ffde3-c6d7-4a62-9f90-6b6529e80df1\") " Oct 29 22:59:01.591681 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.591332 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca" pod "477ffde3-c6d7-4a62-9f90-6b6529e80df1" (UID: "477ffde3-c6d7-4a62-9f90-6b6529e80df1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:01.596280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.594876 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs" pod "477ffde3-c6d7-4a62-9f90-6b6529e80df1" (UID: "477ffde3-c6d7-4a62-9f90-6b6529e80df1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:01.596280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.595288 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls" pod "477ffde3-c6d7-4a62-9f90-6b6529e80df1" (UID: "477ffde3-c6d7-4a62-9f90-6b6529e80df1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:01.596280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.595347 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89" pod "477ffde3-c6d7-4a62-9f90-6b6529e80df1" (UID: "477ffde3-c6d7-4a62-9f90-6b6529e80df1"). InnerVolumeSpecName "kube-api-access-6xn89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:01.693213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.693172 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:01.693213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.693213 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xn89\" (UniqueName: \"kubernetes.io/projected/477ffde3-c6d7-4a62-9f90-6b6529e80df1-kube-api-access-6xn89\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:01.693427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.693229 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/477ffde3-c6d7-4a62-9f90-6b6529e80df1-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:01.693427 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.693243 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/477ffde3-c6d7-4a62-9f90-6b6529e80df1-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:01.944650 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.944601 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh"] Oct 29 22:59:01.949912 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:01.949879 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c7b96a-7444f766c9-bcflh"] Oct 29 22:59:02.839356 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:02.839319 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477ffde3-c6d7-4a62-9f90-6b6529e80df1" path="/var/lib/kubelet/pods/477ffde3-c6d7-4a62-9f90-6b6529e80df1/volumes" Oct 29 22:59:09.256646 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:09.256607 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:59:09.635731 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:09.635619 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x"] Oct 29 22:59:09.635928 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:09.635883 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" containerID="cri-o://78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322" gracePeriod=30 Oct 29 22:59:12.297271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.297234 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" podUID="0fe51827-6a65-4776-b12d-e9bd256b6888" containerName="main" containerID="cri-o://a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7" gracePeriod=45 Oct 29 22:59:12.433026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.432991 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" podUID="0fe51827-6a65-4776-b12d-e9bd256b6888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": dial tcp 10.132.0.160:8000: connect: connection refused" Oct 29 22:59:12.592008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.591981 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:59:12.679925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.679887 5073 generic.go:356] "Generic (PLEG): container finished" podID="0fe51827-6a65-4776-b12d-e9bd256b6888" containerID="a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7" exitCode=0 Oct 29 22:59:12.679925 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.679935 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" Oct 29 22:59:12.680253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.679979 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" event={"ID":"0fe51827-6a65-4776-b12d-e9bd256b6888","Type":"ContainerDied","Data":"a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7"} Oct 29 22:59:12.680253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.680023 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt" event={"ID":"0fe51827-6a65-4776-b12d-e9bd256b6888","Type":"ContainerDied","Data":"ed04bd5531a5404e6ce0b1688fa7f44213c99f2bc9dbd72a5b97b814e8aafc5c"} Oct 29 22:59:12.680253 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.680045 5073 scope.go:122] "RemoveContainer" containerID="a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7" Oct 29 22:59:12.694886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.694864 5073 scope.go:122] "RemoveContainer" containerID="a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7" Oct 29 22:59:12.695244 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:12.695221 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7\": container with ID starting with a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7 not found: ID does not exist" containerID="a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7" Oct 29 22:59:12.695338 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.695257 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7"} err="failed to get container status \"a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7\": rpc error: code = NotFound desc = could not find container \"a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7\": container with ID starting with a44db776169f5daae545f96ab924e70caefb4e0ba16540558d7cc7a35b66c3d7 not found: ID does not exist" Oct 29 22:59:12.725515 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725495 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm\") pod \"0fe51827-6a65-4776-b12d-e9bd256b6888\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " Oct 29 22:59:12.725616 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725582 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx\" (UniqueName: \"kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx\") pod \"0fe51827-6a65-4776-b12d-e9bd256b6888\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " Oct 29 22:59:12.725672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725631 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir\") pod \"0fe51827-6a65-4776-b12d-e9bd256b6888\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " Oct 29 22:59:12.725672 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725654 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache\") pod \"0fe51827-6a65-4776-b12d-e9bd256b6888\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " Oct 29 22:59:12.725750 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725724 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs\") pod \"0fe51827-6a65-4776-b12d-e9bd256b6888\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " Oct 29 22:59:12.725795 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725788 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home\") pod \"0fe51827-6a65-4776-b12d-e9bd256b6888\" (UID: \"0fe51827-6a65-4776-b12d-e9bd256b6888\") " Oct 29 22:59:12.725910 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725883 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir" pod "0fe51827-6a65-4776-b12d-e9bd256b6888" (UID: "0fe51827-6a65-4776-b12d-e9bd256b6888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:12.725991 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.725924 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache" pod "0fe51827-6a65-4776-b12d-e9bd256b6888" (UID: "0fe51827-6a65-4776-b12d-e9bd256b6888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:12.726203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.726092 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home" pod "0fe51827-6a65-4776-b12d-e9bd256b6888" (UID: "0fe51827-6a65-4776-b12d-e9bd256b6888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:12.726623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.726554 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.726623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.726578 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.726623 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.726585 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.728008 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.727983 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx" pod "0fe51827-6a65-4776-b12d-e9bd256b6888" (UID: "0fe51827-6a65-4776-b12d-e9bd256b6888"). InnerVolumeSpecName "kube-api-access-b2mvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:12.728106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.728020 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm" pod "0fe51827-6a65-4776-b12d-e9bd256b6888" (UID: "0fe51827-6a65-4776-b12d-e9bd256b6888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:12.728330 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.728302 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs" pod "0fe51827-6a65-4776-b12d-e9bd256b6888" (UID: "0fe51827-6a65-4776-b12d-e9bd256b6888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:12.828005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.827931 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2mvx\" (UniqueName: \"kubernetes.io/projected/0fe51827-6a65-4776-b12d-e9bd256b6888-kube-api-access-b2mvx\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.828005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.827964 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe51827-6a65-4776-b12d-e9bd256b6888-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.828005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:12.827974 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe51827-6a65-4776-b12d-e9bd256b6888-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:13.023637 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:13.023595 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt"] Oct 29 22:59:13.029039 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:13.029013 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-dbb507/rate-test-model-dbb507-kserve-78685ddc6f-9gbxt"] Oct 29 22:59:14.839404 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:14.839369 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe51827-6a65-4776-b12d-e9bd256b6888" path="/var/lib/kubelet/pods/0fe51827-6a65-4776-b12d-e9bd256b6888/volumes" Oct 29 22:59:14.934032 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:14.934008 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:59:15.063487 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063458 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.063653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063517 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.063653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063541 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.063653 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063581 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.063805 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063651 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.063805 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063693 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.063805 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063759 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:15.064026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063846 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.064026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063891 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:15.064026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063922 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.064026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063975 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:15.064026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.063992 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:15.064026 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064023 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token\") pod \"b6367aa8-5446-423c-992a-2054e486fb65\" (UID: \"b6367aa8-5446-423c-992a-2054e486fb65\") " Oct 29 22:59:15.064385 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064276 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:15.064688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064665 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.064775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064693 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6367aa8-5446-423c-992a-2054e486fb65-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.064775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064703 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.064775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064711 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.064775 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.064718 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.066656 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.066623 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "kube-api-access-srb2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:15.066785 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.066661 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:15.066970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.066941 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:59:15.066970 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.066950 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy" pod "b6367aa8-5446-423c-992a-2054e486fb65" (UID: "b6367aa8-5446-423c-992a-2054e486fb65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:15.165987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.165911 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6367aa8-5446-423c-992a-2054e486fb65-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.165987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.165955 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6367aa8-5446-423c-992a-2054e486fb65-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.165987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.165963 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.165987 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.165972 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srb2b\" (UniqueName: \"kubernetes.io/projected/b6367aa8-5446-423c-992a-2054e486fb65-kube-api-access-srb2b\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.705227 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.705192 5073 generic.go:356] "Generic (PLEG): container finished" podID="b6367aa8-5446-423c-992a-2054e486fb65" containerID="78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322" exitCode=0 Oct 29 22:59:15.705393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.705234 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" Oct 29 22:59:15.705393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.705273 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" event={"ID":"b6367aa8-5446-423c-992a-2054e486fb65","Type":"ContainerDied","Data":"78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322"} Oct 29 22:59:15.705393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.705331 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x" event={"ID":"b6367aa8-5446-423c-992a-2054e486fb65","Type":"ContainerDied","Data":"eaaad47341d682dc504c98dda90bc8a4a4573e87bab40f8e3344ade52cb5c40b"} Oct 29 22:59:15.705393 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.705353 5073 scope.go:122] "RemoveContainer" containerID="78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322" Oct 29 22:59:15.716957 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.716943 5073 scope.go:122] "RemoveContainer" containerID="eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995" Oct 29 22:59:15.726428 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.726410 5073 scope.go:122] "RemoveContainer" containerID="78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322" Oct 29 22:59:15.726687 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:15.726669 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322\": container with ID starting with 78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322 not found: ID does not exist" containerID="78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322" Oct 29 22:59:15.726749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.726695 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322"} err="failed to get container status \"78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322\": rpc error: code = NotFound desc = could not find container \"78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322\": container with ID starting with 78ea80088d199b09135cc40a587677589650e70692560a773f7bba4327408322 not found: ID does not exist" Oct 29 22:59:15.726749 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.726713 5073 scope.go:122] "RemoveContainer" containerID="eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995" Oct 29 22:59:15.726960 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:15.726945 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995\": container with ID starting with eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995 not found: ID does not exist" containerID="eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995" Oct 29 22:59:15.727016 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.726964 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995"} err="failed to get container status \"eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995\": rpc error: code = NotFound desc = could not find container \"eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995\": container with ID starting with eea21d1e93b9b5d35d7d1891a05b8ec7b657c957106df438dc00075c51aee995 not found: ID does not exist" Oct 29 22:59:15.776432 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.776407 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x"] Oct 29 22:59:15.789946 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:15.789926 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c7b96a-openshift-default-766c4f5c5f-h8v5x"] Oct 29 22:59:16.839743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:16.839703 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6367aa8-5446-423c-992a-2054e486fb65" path="/var/lib/kubelet/pods/b6367aa8-5446-423c-992a-2054e486fb65/volumes" Oct 29 22:59:29.521319 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:29.521283 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf"] Oct 29 22:59:29.521900 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:29.521530 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" containerID="cri-o://cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204" gracePeriod=30 Oct 29 22:59:33.549532 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:33.549494 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:59:33.550010 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:33.549590 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:59:34.816621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.816592 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:59:34.872456 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.872420 5073 generic.go:356] "Generic (PLEG): container finished" podID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerID="cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204" exitCode=0 Oct 29 22:59:34.872638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.872471 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" Oct 29 22:59:34.872638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.872509 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" event={"ID":"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda","Type":"ContainerDied","Data":"cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204"} Oct 29 22:59:34.872638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.872577 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf" event={"ID":"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda","Type":"ContainerDied","Data":"ae3eb3d76b9d01d18ab16fd7f4b0bc49a187a048617fd0fced95cf7a87750ccc"} Oct 29 22:59:34.872638 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.872606 5073 scope.go:122] "RemoveContainer" containerID="cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204" Oct 29 22:59:34.885741 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.885722 5073 scope.go:122] "RemoveContainer" containerID="cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb" Oct 29 22:59:34.899374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.899350 5073 scope.go:122] "RemoveContainer" containerID="cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204" Oct 29 22:59:34.899696 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:34.899666 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204\": container with ID starting with cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204 not found: ID does not exist" containerID="cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204" Oct 29 22:59:34.899779 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.899706 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204"} err="failed to get container status \"cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204\": rpc error: code = NotFound desc = could not find container \"cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204\": container with ID starting with cb8e79378f97bb16c9547dfcfcd256480eedda01284e4795a950d8414f3f5204 not found: ID does not exist" Oct 29 22:59:34.899779 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.899725 5073 scope.go:122] "RemoveContainer" containerID="cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb" Oct 29 22:59:34.900007 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:34.899975 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb\": container with ID starting with cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb not found: ID does not exist" containerID="cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb" Oct 29 22:59:34.900086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.900016 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb"} err="failed to get container status \"cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb\": rpc error: code = NotFound desc = could not find container \"cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb\": container with ID starting with cfae57db476bfb7545d8d7d403a9bcedd8b482db06e30dc70cb5d8b7b46875eb not found: ID does not exist" Oct 29 22:59:34.956890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.956861 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.956890 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.956895 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.956958 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957004 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957111 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957078 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957133 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957374 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957321 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957370 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:34.957499 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957448 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957592 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert\") pod \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\" (UID: \"b3461c64-5710-4cd2-bef9-9a3e6ab8aeda\") " Oct 29 22:59:34.957862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957726 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:34.958136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957906 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:34.958136 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.957956 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:34.958317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.958240 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:34.959113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.958711 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:34.959113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.958736 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:34.959113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.958751 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:34.959113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.958788 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:34.959113 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.958809 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:34.960060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.960032 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:34.960324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.960303 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:34.960463 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.960446 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "kube-api-access-hl5hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:34.960601 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.960579 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo" pod "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" (UID: "b3461c64-5710-4cd2-bef9-9a3e6ab8aeda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:59:34.979933 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:34.979904 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf"] Oct 29 22:59:35.002692 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.002665 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx"] Oct 29 22:59:35.060067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.059986 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:35.060067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.060020 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:35.060067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.060032 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5hv\" (UniqueName: \"kubernetes.io/projected/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-kube-api-access-hl5hv\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:35.060067 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.060040 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:35.226867 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.226830 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf"] Oct 29 22:59:35.250242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:35.250199 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e526ec-openshift-default-6ffbb88df-wlgwf"] Oct 29 22:59:36.840286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:36.840252 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" path="/var/lib/kubelet/pods/b3461c64-5710-4cd2-bef9-9a3e6ab8aeda/volumes" Oct 29 22:59:39.256314 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:39.256277 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 22:59:42.839346 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.839315 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9"] Oct 29 22:59:42.840088 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840065 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840089 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840114 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0fe51827-6a65-4776-b12d-e9bd256b6888" containerName="main" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840122 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe51827-6a65-4776-b12d-e9bd256b6888" containerName="main" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840135 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840143 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840180 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4ac4a115-a240-459c-9853-d9d11faa6df7" containerName="main" Oct 29 22:59:42.840219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840187 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac4a115-a240-459c-9853-d9d11faa6df7" containerName="main" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840222 5073 cpu_manager.go:414] "containerMap: removing container" podUID="477ffde3-c6d7-4a62-9f90-6b6529e80df1" containerName="maas-api" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840230 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477ffde3-c6d7-4a62-9f90-6b6529e80df1" containerName="maas-api" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840419 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840432 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840474 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="477ffde3-c6d7-4a62-9f90-6b6529e80df1" containerName="maas-api" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840495 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840508 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe51827-6a65-4776-b12d-e9bd256b6888" containerName="main" Oct 29 22:59:42.840575 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.840518 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac4a115-a240-459c-9853-d9d11faa6df7" containerName="main" Oct 29 22:59:42.844281 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.844257 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.861508 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.861480 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9"] Oct 29 22:59:42.871379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.871352 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9a15f4-openshift-default-dockercfg-6sn4c\"" Oct 29 22:59:42.874948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.874919 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8b8t\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.874969 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.874995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875060 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.875014 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.875063 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.875083 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.875224 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875286 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.875277 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.875500 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.875324 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977361 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977319 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977406 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977544 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977491 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8b8t\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977541 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977581 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977614 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977682 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.977862 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977714 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.978249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977887 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.978249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977892 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.978249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.977985 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.978249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.978061 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.978249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.978132 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.978249 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.978235 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.979976 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.979952 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.980379 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.980359 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.999133 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.999111 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:42.999489 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:42.999467 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8b8t\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:43.158809 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.158721 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:43.486254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.486227 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9"] Oct 29 22:59:43.489160 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:59:43.489117 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04ef323_4e21_490d_a784_4e71f002b17a.slice/crio-ee9a155028841bd1f555a604e6f570e7497754de2dd31db1a3562cc02d246e7a WatchSource:0}: Error finding container ee9a155028841bd1f555a604e6f570e7497754de2dd31db1a3562cc02d246e7a: Status 404 returned error can't find the container with id ee9a155028841bd1f555a604e6f570e7497754de2dd31db1a3562cc02d246e7a Oct 29 22:59:43.492699 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.492657 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:59:43.492814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.492745 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:59:43.492814 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.492791 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:59:43.940797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.940628 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" event={"ID":"d04ef323-4e21-490d-a784-4e71f002b17a","Type":"ContainerStarted","Data":"0d904883d64207714db50ccb9ff816abb37e25c75ad70258eb3f2d66fc746788"} Oct 29 22:59:43.940797 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.940705 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" event={"ID":"d04ef323-4e21-490d-a784-4e71f002b17a","Type":"ContainerStarted","Data":"ee9a155028841bd1f555a604e6f570e7497754de2dd31db1a3562cc02d246e7a"} Oct 29 22:59:43.990178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:43.990098 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podStartSLOduration=1.990078144 podStartE2EDuration="1.990078144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:43.987108189 +0000 UTC m=+2699.302502270" watchObservedRunningTime="2026-10-29 22:59:43.990078144 +0000 UTC m=+2699.305472225" Oct 29 22:59:44.159844 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.159809 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:44.161106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.161082 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:44.161254 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.161133 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:44.896688 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.896655 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx"] Oct 29 22:59:44.897300 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.897281 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" Oct 29 22:59:44.897351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.897304 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3461c64-5710-4cd2-bef9-9a3e6ab8aeda" containerName="istio-proxy" Oct 29 22:59:44.897351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.897335 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" Oct 29 22:59:44.897351 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.897343 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" Oct 29 22:59:44.897569 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.897555 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6367aa8-5446-423c-992a-2054e486fb65" containerName="istio-proxy" Oct 29 22:59:44.901070 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.901054 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:44.933175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.929491 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-329f7b-openshift-default-dockercfg-fvqhf\"" Oct 29 22:59:44.949609 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:44.949574 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx"] Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008541 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008621 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008656 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008769 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008795 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008832 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008882 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008904 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dhw\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.028176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.008940 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134105 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134068 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134130 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134215 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134277 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134294 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134317 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134368 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134364 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134379 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dhw\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134406 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134580 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134743 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134660 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134738 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.134948 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.134922 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.135530 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.135505 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.136880 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.136858 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.137474 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.137454 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.157865 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.157802 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.158018 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.157997 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dhw\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.159922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.159890 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:45.160023 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.159948 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:45.268454 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.268420 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-329f7b-openshift-default-dockercfg-fvqhf\"" Oct 29 22:59:45.273535 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.273514 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:45.387392 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.387353 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh"] Oct 29 22:59:45.393236 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.393208 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.430203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.430104 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-14b0c3-serving-cert\"" Oct 29 22:59:45.446512 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.446478 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.446747 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.446725 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.446854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.446836 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk85g\" (UniqueName: \"kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.446944 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.446926 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.449471 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.449445 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh"] Oct 29 22:59:45.548091 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.548054 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.548313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.548132 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk85g\" (UniqueName: \"kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.548313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.548238 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.548313 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.548307 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.548508 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:45.548382 5073 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-serving-cert: secret "maas-api-e2e-ipp-a-14b0c3-serving-cert" not found Oct 29 22:59:45.548508 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:45.548470 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls podName:2cc50559-210a-4f52-af82-4f2b78a28bcb nodeName:}" failed. No retries permitted until 2026-10-29 22:59:46.048447776 +0000 UTC m=+2701.363841835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls") pod "maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" (UID: "2cc50559-210a-4f52-af82-4f2b78a28bcb") : secret "maas-api-e2e-ipp-a-14b0c3-serving-cert" not found Oct 29 22:59:45.548988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.548966 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.551290 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.551269 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.577781 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.577749 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk85g\" (UniqueName: \"kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:45.639883 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.639845 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx"] Oct 29 22:59:45.642628 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:59:45.642599 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddf42fd_58e3_46a1_9f57_8bebecf625a5.slice/crio-d034f0dd52254b3881747314cb656e4b09ce19c2adf0ae07bb55f7ae9fc2592b WatchSource:0}: Error finding container d034f0dd52254b3881747314cb656e4b09ce19c2adf0ae07bb55f7ae9fc2592b: Status 404 returned error can't find the container with id d034f0dd52254b3881747314cb656e4b09ce19c2adf0ae07bb55f7ae9fc2592b Oct 29 22:59:45.646590 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.646549 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:59:45.646687 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.646641 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:59:45.646754 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.646704 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:59:45.853061 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.852974 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:59:45.856674 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.856652 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 22:59:45.860219 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.860201 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:59:45.863634 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.863610 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 22:59:45.962317 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.962266 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" event={"ID":"5ddf42fd-58e3-46a1-9f57-8bebecf625a5","Type":"ContainerStarted","Data":"b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a"} Oct 29 22:59:45.962803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:45.962338 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" event={"ID":"5ddf42fd-58e3-46a1-9f57-8bebecf625a5","Type":"ContainerStarted","Data":"d034f0dd52254b3881747314cb656e4b09ce19c2adf0ae07bb55f7ae9fc2592b"} Oct 29 22:59:46.028178 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.008619 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podStartSLOduration=2.008600383 podStartE2EDuration="2.008600383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:46.007761494 +0000 UTC m=+2701.323155575" watchObservedRunningTime="2026-10-29 22:59:46.008600383 +0000 UTC m=+2701.323994483" Oct 29 22:59:46.056242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.056193 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:46.058984 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.058947 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:46.063803 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.063779 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:46.169251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.168912 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:46.169251 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.168972 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:46.282213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.280866 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:46.282213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.281936 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:46.282213 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.281993 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:46.468676 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:59:46.468076 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc50559_210a_4f52_af82_4f2b78a28bcb.slice/crio-260d0ca1d7c42fd2c4cef26f0853b4787cc6a6825cfa025564e9978f5aaa6bd9 WatchSource:0}: Error finding container 260d0ca1d7c42fd2c4cef26f0853b4787cc6a6825cfa025564e9978f5aaa6bd9: Status 404 returned error can't find the container with id 260d0ca1d7c42fd2c4cef26f0853b4787cc6a6825cfa025564e9978f5aaa6bd9 Oct 29 22:59:46.468676 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.468380 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh"] Oct 29 22:59:46.469734 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.469710 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:59:46.974615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.974276 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" event={"ID":"2cc50559-210a-4f52-af82-4f2b78a28bcb","Type":"ContainerStarted","Data":"11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d"} Oct 29 22:59:46.974615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.974336 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" event={"ID":"2cc50559-210a-4f52-af82-4f2b78a28bcb","Type":"ContainerStarted","Data":"260d0ca1d7c42fd2c4cef26f0853b4787cc6a6825cfa025564e9978f5aaa6bd9"} Oct 29 22:59:46.974615 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:46.974422 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:47.070331 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:47.070217 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" podStartSLOduration=1.752887706 podStartE2EDuration="2.070196551s" totalImagesPullingTime="317.308845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:47.043520319 +0000 UTC m=+2702.358914400" watchObservedRunningTime="2026-10-29 22:59:47.070196551 +0000 UTC m=+2702.385590632" Oct 29 22:59:47.161179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:47.160507 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:47.161179 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:47.160570 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:47.274171 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:47.274116 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:47.274455 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:47.274425 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:48.159613 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:48.159577 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:48.160265 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:48.160229 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:48.274293 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:48.274261 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:48.274482 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:48.274310 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:48.669071 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:48.669040 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 22:59:49.163170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:49.160506 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:49.163170 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:49.160566 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:49.274359 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:49.274324 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:49.274551 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:49.274374 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:50.046013 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.041318 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" containerID="cri-o://784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5" gracePeriod=45 Oct 29 22:59:50.067176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.061784 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" containerID="cri-o://e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a" gracePeriod=45 Oct 29 22:59:50.073314 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.073266 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": EOF" Oct 29 22:59:50.159920 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.159880 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:59:50.160084 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.159937 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 22:59:50.282341 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.282304 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:50.282838 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.282372 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:50.688208 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.687858 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:59:50.704907 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.704863 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache\") pod \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " Oct 29 22:59:50.705086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.704943 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs\") pod \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " Oct 29 22:59:50.705086 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.704993 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir\") pod \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " Oct 29 22:59:50.705280 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.705237 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm\") pod \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " Oct 29 22:59:50.705366 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.705346 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk\" (UniqueName: \"kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk\") pod \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " Oct 29 22:59:50.705441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.705381 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home\") pod \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\" (UID: \"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3\") " Oct 29 22:59:50.706327 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.706294 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home" pod "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" (UID: "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.706533 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.706507 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache" pod "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" (UID: "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.708327 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.708303 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:59:50.725504 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.725462 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir" pod "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" (UID: "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.729621 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.729585 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk" pod "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" (UID: "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3"). InnerVolumeSpecName "kube-api-access-v8tzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:50.730196 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.730167 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm" pod "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" (UID: "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.758005 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.755241 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs" pod "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" (UID: "5fd2b64d-24fa-4572-adb7-bfc7ebac61d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.811903 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm\") pod \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.812041 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs\") pod \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.812071 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache\") pod \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.812186 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home\") pod \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.812239 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir\") pod \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.812302 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp\" (UniqueName: \"kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp\") pod \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\" (UID: \"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c\") " Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.812991 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.813014 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.813028 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.813040 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.813053 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tzk\" (UniqueName: \"kubernetes.io/projected/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-kube-api-access-v8tzk\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.813066 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.830175 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.829120 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache" pod "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" (UID: "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.845562 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.845524 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home" pod "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" (UID: "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.845988 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.845965 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir" pod "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" (UID: "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.916911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.916826 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.916911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.916863 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.916911 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.916881 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.931190 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.929350 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm" pod "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" (UID: "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:50.947190 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.945381 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp" pod "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" (UID: "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c"). InnerVolumeSpecName "kube-api-access-4kkqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:50.980176 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:50.977342 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs" pod "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" (UID: "3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:51.055057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.052770 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:51.055057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.052807 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:51.055057 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.052823 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kkqp\" (UniqueName: \"kubernetes.io/projected/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c-kube-api-access-4kkqp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.070256 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6"] Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.071515 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.071535 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.071586 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.071594 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.071809 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" Oct 29 22:59:51.076203 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.071825 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerName="main" Oct 29 22:59:51.086402 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.086364 5073 generic.go:356] "Generic (PLEG): container finished" podID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" containerID="784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5" exitCode=0 Oct 29 22:59:51.086811 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.086775 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" event={"ID":"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c","Type":"ContainerDied","Data":"784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5"} Oct 29 22:59:51.086922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.086837 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" event={"ID":"3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c","Type":"ContainerDied","Data":"48812a30a11338acecb87e7c605e5836581247f84325fb02cfb9a03a6a8c061b"} Oct 29 22:59:51.086922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.086863 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6"] Oct 29 22:59:51.086922 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.086914 5073 scope.go:122] "RemoveContainer" containerID="784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5" Oct 29 22:59:51.087094 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.086992 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.087292 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.087274 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf" Oct 29 22:59:51.090614 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.090588 5073 generic.go:356] "Generic (PLEG): container finished" podID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerID="e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a" exitCode=0 Oct 29 22:59:51.090919 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.090900 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" event={"ID":"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3","Type":"ContainerDied","Data":"e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a"} Oct 29 22:59:51.092886 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.092858 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" event={"ID":"5fd2b64d-24fa-4572-adb7-bfc7ebac61d3","Type":"ContainerDied","Data":"4bbc86061c211551958624521f829a139d8e952405f16e68aa15500534a1c252"} Oct 29 22:59:51.093106 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.093091 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" Oct 29 22:59:51.104073 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.104043 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bcb810\"" Oct 29 22:59:51.104705 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.104682 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bcb810-dockercfg-zzzr2\"" Oct 29 22:59:51.151324 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.151209 5073 scope.go:122] "RemoveContainer" containerID="784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5" Oct 29 22:59:51.152109 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:51.152074 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5\": container with ID starting with 784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5 not found: ID does not exist" containerID="784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5" Oct 29 22:59:51.152369 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.152337 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5"} err="failed to get container status \"784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5\": rpc error: code = NotFound desc = could not find container \"784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5\": container with ID starting with 784d304c893a02451a5ab6f3f3923dc327a73cd392e8326333f4d60ee208b4e5 not found: ID does not exist" Oct 29 22:59:51.152501 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.152487 5073 scope.go:122] "RemoveContainer" containerID="e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a" Oct 29 22:59:51.157258 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.157169 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.157390 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.157295 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshqm\" (UniqueName: \"kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm\") pod \"payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.170440 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.169778 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:51.172030 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.170938 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:51.173217 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.172209 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 22:59:51.174893 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.174842 5073 scope.go:122] "RemoveContainer" containerID="e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a" Oct 29 22:59:51.175283 ip-10-0-134-94 kubenswrapper[5073]: E1029 22:59:51.175260 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a\": container with ID starting with e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a not found: ID does not exist" containerID="e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a" Oct 29 22:59:51.175364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.175290 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a"} err="failed to get container status \"e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a\": rpc error: code = NotFound desc = could not find container \"e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a\": container with ID starting with e542888d424a4a25d778cf9a4e94bfc52eb4a32d84174cd977aea411e88d4c3a not found: ID does not exist" Oct 29 22:59:51.258494 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.258419 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshqm\" (UniqueName: \"kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm\") pod \"payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.258762 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.258700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.259423 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.259403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.273916 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.273885 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:51.274075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.273937 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:51.290271 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.290238 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshqm\" (UniqueName: \"kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm\") pod \"payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.349242 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.349205 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf"] Oct 29 22:59:51.352854 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.352822 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/distinct-w0-40cf9a-kserve-dd7858b54-vvnmf"] Oct 29 22:59:51.394953 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.394920 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx"] Oct 29 22:59:51.406983 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.406948 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx"] Oct 29 22:59:51.468858 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.468826 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:51.490024 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.489986 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-40cf9a/simulator-w0-40cf9a-kserve-589ff674d6-cqdxx" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:51.836116 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:51.836029 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6"] Oct 29 22:59:51.839994 ip-10-0-134-94 kubenswrapper[5073]: W1029 22:59:51.839953 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c8d02e_aa88_4237_bc6e_d18b28ffaeb6.slice/crio-2b3fee2e989c1fe5af18757cf7330e1addcc9d3d5b664fe76358ed812b267f6b WatchSource:0}: Error finding container 2b3fee2e989c1fe5af18757cf7330e1addcc9d3d5b664fe76358ed812b267f6b: Status 404 returned error can't find the container with id 2b3fee2e989c1fe5af18757cf7330e1addcc9d3d5b664fe76358ed812b267f6b Oct 29 22:59:52.102014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.101914 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" event={"ID":"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6","Type":"ContainerStarted","Data":"58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d"} Oct 29 22:59:52.102014 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.101979 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" event={"ID":"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6","Type":"ContainerStarted","Data":"2b3fee2e989c1fe5af18757cf7330e1addcc9d3d5b664fe76358ed812b267f6b"} Oct 29 22:59:52.102237 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.102133 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 22:59:52.150364 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.150310 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" podStartSLOduration=2.150296494 podStartE2EDuration="2.150296494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:52.150093555 +0000 UTC m=+2707.465487640" watchObservedRunningTime="2026-10-29 22:59:52.150296494 +0000 UTC m=+2707.465690574" Oct 29 22:59:52.276777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.274987 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:59:52.276777 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.275086 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 22:59:52.845386 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.845350 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c" path="/var/lib/kubelet/pods/3ce1d8ec-3c00-405f-a6d0-d5e520cc2d4c/volumes" Oct 29 22:59:52.846075 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.846052 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd2b64d-24fa-4572-adb7-bfc7ebac61d3" path="/var/lib/kubelet/pods/5fd2b64d-24fa-4572-adb7-bfc7ebac61d3/volumes" Oct 29 22:59:52.987441 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:52.987410 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 22:59:53.281284 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:53.281251 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:53.281768 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:53.281743 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 22:59:53.282585 ip-10-0-134-94 kubenswrapper[5073]: I1029 22:59:53.282556 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 23:00:00.273893 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.273836 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr"] Oct 29 23:00:00.286922 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.286889 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj"] Oct 29 23:00:00.290462 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.290433 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.293085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.293049 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.315130 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.315098 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj"] Oct 29 23:00:00.318852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.318803 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr"] Oct 29 23:00:00.322658 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.322630 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:00:00.325540 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.325515 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw"] Oct 29 23:00:00.332949 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.332923 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q"] Oct 29 23:00:00.335411 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.335389 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.338100 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.338075 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5"] Oct 29 23:00:00.338286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.338264 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.343992 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.343473 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.360277 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.360215 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q"] Oct 29 23:00:00.365556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.365538 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw"] Oct 29 23:00:00.373843 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.373815 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n"] Oct 29 23:00:00.379514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.379489 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.381646 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.381451 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5"] Oct 29 23:00:00.409364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.407886 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x77tk\" (UniqueName: \"kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.409364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.407959 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.409364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.408006 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwwp\" (UniqueName: \"kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.409364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.408224 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.414178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.411970 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n"] Oct 29 23:00:00.509215 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509138 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.509404 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509283 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqz8\" (UniqueName: \"kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8\") pod \"maas-api-key-cleanup-29888580-q8rm5\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.509404 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509386 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.509522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509425 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrgl\" (UniqueName: \"kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl\") pod \"maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.509522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509463 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsrw\" (UniqueName: \"kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.509522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509512 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.509671 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509580 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.509671 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509633 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-q8rm5\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.509778 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509755 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x77tk\" (UniqueName: \"kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.509831 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509819 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.509997 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.509975 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5sd\" (UniqueName: \"kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.510088 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.510034 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwwp\" (UniqueName: \"kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.510163 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.510088 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.510552 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.510521 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.531940 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.531871 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwwp\" (UniqueName: \"kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.531940 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.531875 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x77tk\" (UniqueName: \"kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.611133 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611089 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5sd\" (UniqueName: \"kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611196 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611233 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqz8\" (UniqueName: \"kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8\") pod \"maas-api-key-cleanup-29888580-q8rm5\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611285 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrgl\" (UniqueName: \"kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl\") pod \"maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611311 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsrw\" (UniqueName: \"kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611354 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611380 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.611449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611421 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-q8rm5\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.611974 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.611951 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.612067 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.612049 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-q8rm5\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.612204 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.612181 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.612339 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.612324 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.630387 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.630353 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:00.633184 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.633132 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqz8\" (UniqueName: \"kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8\") pod \"maas-api-key-cleanup-29888580-q8rm5\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.633184 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.633164 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrgl\" (UniqueName: \"kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl\") pod \"maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.633714 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.633252 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5sd\" (UniqueName: \"kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.633714 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.633256 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsrw\" (UniqueName: \"kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw\") pod \"maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.645517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.645398 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:00.666853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.666630 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:00.674713 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.674438 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:00.687191 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.687160 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:00.704875 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.704842 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:00.948207 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:00.948179 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr"] Oct 29 23:00:00.948589 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:00.948564 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e7e415_e2ea_4cdd_adf5_784df46f6589.slice/crio-13ddcc281a525ce6dfffdec114de9064cef7cdfcb73df06f99a7efe0a53f3c97 WatchSource:0}: Error finding container 13ddcc281a525ce6dfffdec114de9064cef7cdfcb73df06f99a7efe0a53f3c97: Status 404 returned error can't find the container with id 13ddcc281a525ce6dfffdec114de9064cef7cdfcb73df06f99a7efe0a53f3c97 Oct 29 23:00:01.018482 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.017671 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw"] Oct 29 23:00:01.077453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.077430 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n"] Oct 29 23:00:01.080737 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.080714 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q"] Oct 29 23:00:01.106603 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.106570 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5"] Oct 29 23:00:01.111129 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:01.111098 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156fdc41_dcc8_4740_843f_bbe96939f631.slice/crio-1926e9cf5294e71ab44cf23ecfbb05b812e13f89b86dea82929d7733d9c27c7b WatchSource:0}: Error finding container 1926e9cf5294e71ab44cf23ecfbb05b812e13f89b86dea82929d7733d9c27c7b: Status 404 returned error can't find the container with id 1926e9cf5294e71ab44cf23ecfbb05b812e13f89b86dea82929d7733d9c27c7b Oct 29 23:00:01.111941 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:01.111914 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbe33e6_8de8_447e_aef4_d52ca589bc46.slice/crio-8e13c803e3f3f7a696ce89152d5be5f7d321cc86e5d98ccfa49c8840508bedec WatchSource:0}: Error finding container 8e13c803e3f3f7a696ce89152d5be5f7d321cc86e5d98ccfa49c8840508bedec: Status 404 returned error can't find the container with id 8e13c803e3f3f7a696ce89152d5be5f7d321cc86e5d98ccfa49c8840508bedec Oct 29 23:00:01.113303 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:01.113270 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81979462_3286_41c9_8c4e_24e619e94c36.slice/crio-3bf0076682b7a0716b40843c56bc4829f6e74b0913b16d8a6923d2353eb1bdee WatchSource:0}: Error finding container 3bf0076682b7a0716b40843c56bc4829f6e74b0913b16d8a6923d2353eb1bdee: Status 404 returned error can't find the container with id 3bf0076682b7a0716b40843c56bc4829f6e74b0913b16d8a6923d2353eb1bdee Oct 29 23:00:01.115203 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:01.115177 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261794cc_16dc_4dea_8182_0ba53b471a80.slice/crio-140617112f5ae55dc08a3f7e7a04895d5f4352c8bf586ceb50cb15446d99bec9 WatchSource:0}: Error finding container 140617112f5ae55dc08a3f7e7a04895d5f4352c8bf586ceb50cb15446d99bec9: Status 404 returned error can't find the container with id 140617112f5ae55dc08a3f7e7a04895d5f4352c8bf586ceb50cb15446d99bec9 Oct 29 23:00:01.171719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.171684 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerStarted","Data":"3bf0076682b7a0716b40843c56bc4829f6e74b0913b16d8a6923d2353eb1bdee"} Oct 29 23:00:01.173463 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.173436 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerStarted","Data":"8e13c803e3f3f7a696ce89152d5be5f7d321cc86e5d98ccfa49c8840508bedec"} Oct 29 23:00:01.174162 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:01.174113 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942842bf_be49_40ae_ae97_674faaf23a6e.slice/crio-c4774f9cb40cf81fe8b738067ef6f6b9a7ccd49d439d713a5711970e70e5f40c WatchSource:0}: Error finding container c4774f9cb40cf81fe8b738067ef6f6b9a7ccd49d439d713a5711970e70e5f40c: Status 404 returned error can't find the container with id c4774f9cb40cf81fe8b738067ef6f6b9a7ccd49d439d713a5711970e70e5f40c Oct 29 23:00:01.174886 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.174860 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj"] Oct 29 23:00:01.180300 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.180272 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerStarted","Data":"140617112f5ae55dc08a3f7e7a04895d5f4352c8bf586ceb50cb15446d99bec9"} Oct 29 23:00:01.183493 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.183465 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerStarted","Data":"1926e9cf5294e71ab44cf23ecfbb05b812e13f89b86dea82929d7733d9c27c7b"} Oct 29 23:00:01.184546 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:01.184523 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerStarted","Data":"13ddcc281a525ce6dfffdec114de9064cef7cdfcb73df06f99a7efe0a53f3c97"} Oct 29 23:00:02.194730 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.194687 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerStarted","Data":"dc380901117fb2d23386604e483a61b2da23b8b9949cf304fe8ba88c76116993"} Oct 29 23:00:02.196438 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.196410 5073 generic.go:356] "Generic (PLEG): container finished" podID="156fdc41-dcc8-4740-843f-bbe96939f631" containerID="373f844ae706faa158117bfcffda4af7b5fa570c31bfebd8eb0d35906d8d7192" exitCode=60 Oct 29 23:00:02.196555 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.196501 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerDied","Data":"373f844ae706faa158117bfcffda4af7b5fa570c31bfebd8eb0d35906d8d7192"} Oct 29 23:00:02.196748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.196729 5073 scope.go:122] "RemoveContainer" containerID="373f844ae706faa158117bfcffda4af7b5fa570c31bfebd8eb0d35906d8d7192" Oct 29 23:00:02.198501 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.198476 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerStarted","Data":"92a91118c9b3a9f91ebeef172d635fc5c3fdfc9a3eb88d024645e7f3fb6e9ad0"} Oct 29 23:00:02.200678 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.200658 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerStarted","Data":"65c9cd24fd3763f372985564b0abfc6dc1335cb44b290c0c5617b4b508a6386c"} Oct 29 23:00:02.202469 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.202445 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerStarted","Data":"9678f2148b9240ef0cd1ec70ab465bc84a46a2b0eb9f6ee5ac887d19a3a83c82"} Oct 29 23:00:02.204075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.204049 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerStarted","Data":"1c16904bd4432488c33b0808736009a3efada6eb0f64e0024e57f828f5755842"} Oct 29 23:00:02.204196 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.204094 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerStarted","Data":"c4774f9cb40cf81fe8b738067ef6f6b9a7ccd49d439d713a5711970e70e5f40c"} Oct 29 23:00:02.250503 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.250410 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" podStartSLOduration=2.250391518 podStartE2EDuration="2.250391518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.241540217 +0000 UTC m=+2717.556934330" watchObservedRunningTime="2026-10-29 23:00:02.250391518 +0000 UTC m=+2717.565785599" Oct 29 23:00:02.285389 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.285303 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" podStartSLOduration=2.285287038 podStartE2EDuration="2.285287038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.282433751 +0000 UTC m=+2717.597827836" watchObservedRunningTime="2026-10-29 23:00:02.285287038 +0000 UTC m=+2717.600681118" Oct 29 23:00:02.376501 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.376390 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" podStartSLOduration=2.376375819 podStartE2EDuration="2.376375819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.373039095 +0000 UTC m=+2717.688433185" watchObservedRunningTime="2026-10-29 23:00:02.376375819 +0000 UTC m=+2717.691769898" Oct 29 23:00:02.418032 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.417984 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" podStartSLOduration=2.417970467 podStartE2EDuration="2.417970467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.414927009 +0000 UTC m=+2717.730321089" watchObservedRunningTime="2026-10-29 23:00:02.417970467 +0000 UTC m=+2717.733364546" Oct 29 23:00:02.456515 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:02.456464 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" podStartSLOduration=2.456449565 podStartE2EDuration="2.456449565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.45504404 +0000 UTC m=+2717.770438120" watchObservedRunningTime="2026-10-29 23:00:02.456449565 +0000 UTC m=+2717.771843645" Oct 29 23:00:02.519850 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:02.519816 5073 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e7e415_e2ea_4cdd_adf5_784df46f6589.slice/crio-92a91118c9b3a9f91ebeef172d635fc5c3fdfc9a3eb88d024645e7f3fb6e9ad0.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:00:03.114865 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.114553 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 23:00:03.213182 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.213109 5073 generic.go:356] "Generic (PLEG): container finished" podID="81979462-3286-41c9-8c4e-24e619e94c36" containerID="65c9cd24fd3763f372985564b0abfc6dc1335cb44b290c0c5617b4b508a6386c" exitCode=60 Oct 29 23:00:03.213663 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.213192 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerDied","Data":"65c9cd24fd3763f372985564b0abfc6dc1335cb44b290c0c5617b4b508a6386c"} Oct 29 23:00:03.213663 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.213580 5073 scope.go:122] "RemoveContainer" containerID="65c9cd24fd3763f372985564b0abfc6dc1335cb44b290c0c5617b4b508a6386c" Oct 29 23:00:03.216001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.215976 5073 generic.go:356] "Generic (PLEG): container finished" podID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerID="9678f2148b9240ef0cd1ec70ab465bc84a46a2b0eb9f6ee5ac887d19a3a83c82" exitCode=60 Oct 29 23:00:03.216125 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.216066 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerDied","Data":"9678f2148b9240ef0cd1ec70ab465bc84a46a2b0eb9f6ee5ac887d19a3a83c82"} Oct 29 23:00:03.216380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.216363 5073 scope.go:122] "RemoveContainer" containerID="9678f2148b9240ef0cd1ec70ab465bc84a46a2b0eb9f6ee5ac887d19a3a83c82" Oct 29 23:00:03.218001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.217981 5073 generic.go:356] "Generic (PLEG): container finished" podID="942842bf-be49-40ae-ae97-674faaf23a6e" containerID="1c16904bd4432488c33b0808736009a3efada6eb0f64e0024e57f828f5755842" exitCode=60 Oct 29 23:00:03.218130 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.218069 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerDied","Data":"1c16904bd4432488c33b0808736009a3efada6eb0f64e0024e57f828f5755842"} Oct 29 23:00:03.218548 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.218533 5073 scope.go:122] "RemoveContainer" containerID="1c16904bd4432488c33b0808736009a3efada6eb0f64e0024e57f828f5755842" Oct 29 23:00:03.220409 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.220386 5073 generic.go:356] "Generic (PLEG): container finished" podID="261794cc-16dc-4dea-8182-0ba53b471a80" containerID="dc380901117fb2d23386604e483a61b2da23b8b9949cf304fe8ba88c76116993" exitCode=60 Oct 29 23:00:03.220536 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.220471 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerDied","Data":"dc380901117fb2d23386604e483a61b2da23b8b9949cf304fe8ba88c76116993"} Oct 29 23:00:03.220677 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.220656 5073 scope.go:122] "RemoveContainer" containerID="dc380901117fb2d23386604e483a61b2da23b8b9949cf304fe8ba88c76116993" Oct 29 23:00:03.223753 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.223730 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerStarted","Data":"782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e"} Oct 29 23:00:03.226764 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.226739 5073 generic.go:356] "Generic (PLEG): container finished" podID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerID="92a91118c9b3a9f91ebeef172d635fc5c3fdfc9a3eb88d024645e7f3fb6e9ad0" exitCode=60 Oct 29 23:00:03.226881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.226801 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerDied","Data":"92a91118c9b3a9f91ebeef172d635fc5c3fdfc9a3eb88d024645e7f3fb6e9ad0"} Oct 29 23:00:03.227313 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.227208 5073 scope.go:122] "RemoveContainer" containerID="92a91118c9b3a9f91ebeef172d635fc5c3fdfc9a3eb88d024645e7f3fb6e9ad0" Oct 29 23:00:03.383093 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:03.383001 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" podStartSLOduration=3.382987305 podStartE2EDuration="3.382987305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.380565474 +0000 UTC m=+2718.695959551" watchObservedRunningTime="2026-10-29 23:00:03.382987305 +0000 UTC m=+2718.698381384" Oct 29 23:00:04.235515 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.235479 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerStarted","Data":"295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69"} Oct 29 23:00:04.238302 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.238272 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerStarted","Data":"13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c"} Oct 29 23:00:04.240332 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.240303 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerStarted","Data":"b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f"} Oct 29 23:00:04.242556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.242530 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerStarted","Data":"eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69"} Oct 29 23:00:04.244454 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.244429 5073 generic.go:356] "Generic (PLEG): container finished" podID="156fdc41-dcc8-4740-843f-bbe96939f631" containerID="782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e" exitCode=60 Oct 29 23:00:04.244571 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.244452 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerDied","Data":"782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e"} Oct 29 23:00:04.244571 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.244501 5073 scope.go:122] "RemoveContainer" containerID="373f844ae706faa158117bfcffda4af7b5fa570c31bfebd8eb0d35906d8d7192" Oct 29 23:00:04.244713 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.244697 5073 scope.go:122] "RemoveContainer" containerID="782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e" Oct 29 23:00:04.244999 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:04.244976 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw_odh-ai-gateway-infra(156fdc41-dcc8-4740-843f-bbe96939f631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" Oct 29 23:00:04.246594 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.246570 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerStarted","Data":"6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2"} Oct 29 23:00:04.424910 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.424880 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7"] Oct 29 23:00:04.429460 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.429438 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.446022 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.445992 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:04.448604 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.448579 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7"] Oct 29 23:00:04.474929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.474904 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4p9m\" (UniqueName: \"kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.475135 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.475114 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.475274 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.475248 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.576093 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.576066 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.576234 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.576175 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.576359 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.576344 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4p9m\" (UniqueName: \"kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.576814 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.576796 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.578477 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.578460 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.598947 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.598920 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4p9m\" (UniqueName: \"kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m\") pod \"maas-api-revoke-sub-f04dbbc3-796w7\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:04.792607 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:04.792535 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:05.100050 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:05.100015 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4eb895b_52a2_419d_9067_5556244daebe.slice/crio-4850e6a1c997332d573ec08e60b0c097bbd327c4b34b166f36eb568196459369 WatchSource:0}: Error finding container 4850e6a1c997332d573ec08e60b0c097bbd327c4b34b166f36eb568196459369: Status 404 returned error can't find the container with id 4850e6a1c997332d573ec08e60b0c097bbd327c4b34b166f36eb568196459369 Oct 29 23:00:05.100050 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.100044 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7"] Oct 29 23:00:05.137214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.137185 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272"] Oct 29 23:00:05.142871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.142851 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.162926 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.162893 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272"] Oct 29 23:00:05.191493 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.191463 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7w2\" (UniqueName: \"kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.191694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.191506 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.191694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.191543 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.257547 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.257509 5073 generic.go:356] "Generic (PLEG): container finished" podID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerID="13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c" exitCode=60 Oct 29 23:00:05.258037 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.257590 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerDied","Data":"13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c"} Oct 29 23:00:05.258037 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.257657 5073 scope.go:122] "RemoveContainer" containerID="9678f2148b9240ef0cd1ec70ab465bc84a46a2b0eb9f6ee5ac887d19a3a83c82" Oct 29 23:00:05.258037 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.258013 5073 scope.go:122] "RemoveContainer" containerID="13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c" Oct 29 23:00:05.258461 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:05.258423 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n_odh-ai-gateway-infra(cdbe33e6-8de8-447e-aef4-d52ca589bc46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" Oct 29 23:00:05.259746 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.259724 5073 generic.go:356] "Generic (PLEG): container finished" podID="942842bf-be49-40ae-ae97-674faaf23a6e" containerID="b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f" exitCode=60 Oct 29 23:00:05.259855 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.259774 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerDied","Data":"b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f"} Oct 29 23:00:05.260078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.260063 5073 scope.go:122] "RemoveContainer" containerID="b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f" Oct 29 23:00:05.260460 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:05.260432 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj_odh-ai-gateway-infra(942842bf-be49-40ae-ae97-674faaf23a6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" Oct 29 23:00:05.261643 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.261623 5073 generic.go:356] "Generic (PLEG): container finished" podID="261794cc-16dc-4dea-8182-0ba53b471a80" containerID="eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69" exitCode=60 Oct 29 23:00:05.261725 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.261704 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerDied","Data":"eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69"} Oct 29 23:00:05.261970 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.261953 5073 scope.go:122] "RemoveContainer" containerID="eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69" Oct 29 23:00:05.262357 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:05.262334 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-q8rm5_odh-ai-gateway-infra(261794cc-16dc-4dea-8182-0ba53b471a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" Oct 29 23:00:05.263976 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.263946 5073 scope.go:122] "RemoveContainer" containerID="782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e" Oct 29 23:00:05.264248 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:05.264224 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw_odh-ai-gateway-infra(156fdc41-dcc8-4740-843f-bbe96939f631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" Oct 29 23:00:05.265745 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.265722 5073 generic.go:356] "Generic (PLEG): container finished" podID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerID="6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2" exitCode=60 Oct 29 23:00:05.265837 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.265800 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerDied","Data":"6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2"} Oct 29 23:00:05.266076 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.266056 5073 scope.go:122] "RemoveContainer" containerID="6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2" Oct 29 23:00:05.266379 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:05.266351 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr_odh-ai-gateway-infra(87e7e415-e2ea-4cdd-adf5-784df46f6589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" Oct 29 23:00:05.268100 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.268076 5073 generic.go:356] "Generic (PLEG): container finished" podID="81979462-3286-41c9-8c4e-24e619e94c36" containerID="295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69" exitCode=60 Oct 29 23:00:05.268219 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.268097 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerDied","Data":"295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69"} Oct 29 23:00:05.268434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.268415 5073 scope.go:122] "RemoveContainer" containerID="295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69" Oct 29 23:00:05.268803 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:05.268772 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q_odh-ai-gateway-infra(81979462-3286-41c9-8c4e-24e619e94c36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" podUID="81979462-3286-41c9-8c4e-24e619e94c36" Oct 29 23:00:05.269769 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.269749 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" event={"ID":"c4eb895b-52a2-419d-9067-5556244daebe","Type":"ContainerStarted","Data":"4850e6a1c997332d573ec08e60b0c097bbd327c4b34b166f36eb568196459369"} Oct 29 23:00:05.272734 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.272715 5073 scope.go:122] "RemoveContainer" containerID="1c16904bd4432488c33b0808736009a3efada6eb0f64e0024e57f828f5755842" Oct 29 23:00:05.286665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.286639 5073 scope.go:122] "RemoveContainer" containerID="dc380901117fb2d23386604e483a61b2da23b8b9949cf304fe8ba88c76116993" Oct 29 23:00:05.293377 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.293351 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.293976 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.293826 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7w2\" (UniqueName: \"kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.293976 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.293895 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.294183 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.294053 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.301479 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.301457 5073 scope.go:122] "RemoveContainer" containerID="92a91118c9b3a9f91ebeef172d635fc5c3fdfc9a3eb88d024645e7f3fb6e9ad0" Oct 29 23:00:05.308985 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.308956 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.317013 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.316985 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7w2\" (UniqueName: \"kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2\") pod \"maas-api-revoke-sub-7187784e-rn272\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.319517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.319496 5073 scope.go:122] "RemoveContainer" containerID="65c9cd24fd3763f372985564b0abfc6dc1335cb44b290c0c5617b4b508a6386c" Oct 29 23:00:05.458718 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.458689 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:05.782558 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:05.782515 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272"] Oct 29 23:00:05.782912 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:05.782886 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25a87b5_aab0_4a48_9269_ab734ade47c7.slice/crio-2378bb61583a0ab30d3a7aef973b5761999b283497ae0bfa65150eff9ae17769 WatchSource:0}: Error finding container 2378bb61583a0ab30d3a7aef973b5761999b283497ae0bfa65150eff9ae17769: Status 404 returned error can't find the container with id 2378bb61583a0ab30d3a7aef973b5761999b283497ae0bfa65150eff9ae17769 Oct 29 23:00:06.282589 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.282516 5073 scope.go:122] "RemoveContainer" containerID="295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69" Oct 29 23:00:06.283018 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:06.282942 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q_odh-ai-gateway-infra(81979462-3286-41c9-8c4e-24e619e94c36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" podUID="81979462-3286-41c9-8c4e-24e619e94c36" Oct 29 23:00:06.284252 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.284228 5073 generic.go:356] "Generic (PLEG): container finished" podID="c4eb895b-52a2-419d-9067-5556244daebe" containerID="6a72c9bb8971c8359f5ba7ef84776a49e96dd324df53b466f9f7811d6dd307b5" exitCode=0 Oct 29 23:00:06.284359 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.284312 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" event={"ID":"c4eb895b-52a2-419d-9067-5556244daebe","Type":"ContainerDied","Data":"6a72c9bb8971c8359f5ba7ef84776a49e96dd324df53b466f9f7811d6dd307b5"} Oct 29 23:00:06.286481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.286456 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" event={"ID":"d25a87b5-aab0-4a48-9269-ab734ade47c7","Type":"ContainerStarted","Data":"9ec61e0c5a0ae0f4b52f02b4e5c72e8e57f1230bd3a23dc855c9890bca4fc105"} Oct 29 23:00:06.286613 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.286501 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" event={"ID":"d25a87b5-aab0-4a48-9269-ab734ade47c7","Type":"ContainerStarted","Data":"2378bb61583a0ab30d3a7aef973b5761999b283497ae0bfa65150eff9ae17769"} Oct 29 23:00:06.288550 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.288530 5073 scope.go:122] "RemoveContainer" containerID="13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c" Oct 29 23:00:06.288909 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:06.288877 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n_odh-ai-gateway-infra(cdbe33e6-8de8-447e-aef4-d52ca589bc46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" Oct 29 23:00:06.291034 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.291011 5073 scope.go:122] "RemoveContainer" containerID="b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f" Oct 29 23:00:06.291835 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:06.291804 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj_odh-ai-gateway-infra(942842bf-be49-40ae-ae97-674faaf23a6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" Oct 29 23:00:06.325248 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.324532 5073 scope.go:122] "RemoveContainer" containerID="eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69" Oct 29 23:00:06.325629 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:06.325550 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-q8rm5_odh-ai-gateway-infra(261794cc-16dc-4dea-8182-0ba53b471a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" Oct 29 23:00:06.348256 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.348225 5073 scope.go:122] "RemoveContainer" containerID="6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2" Oct 29 23:00:06.349002 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:06.348962 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr_odh-ai-gateway-infra(87e7e415-e2ea-4cdd-adf5-784df46f6589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" Oct 29 23:00:06.514829 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:06.514766 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" podStartSLOduration=1.514751848 podStartE2EDuration="1.514751848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:06.512458553 +0000 UTC m=+2721.827852633" watchObservedRunningTime="2026-10-29 23:00:06.514751848 +0000 UTC m=+2721.830145932" Oct 29 23:00:07.368565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.361336 5073 generic.go:356] "Generic (PLEG): container finished" podID="d25a87b5-aab0-4a48-9269-ab734ade47c7" containerID="9ec61e0c5a0ae0f4b52f02b4e5c72e8e57f1230bd3a23dc855c9890bca4fc105" exitCode=0 Oct 29 23:00:07.368565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.361706 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" event={"ID":"d25a87b5-aab0-4a48-9269-ab734ade47c7","Type":"ContainerDied","Data":"9ec61e0c5a0ae0f4b52f02b4e5c72e8e57f1230bd3a23dc855c9890bca4fc105"} Oct 29 23:00:07.577126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.577094 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:07.642899 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.642806 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca\") pod \"c4eb895b-52a2-419d-9067-5556244daebe\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " Oct 29 23:00:07.643082 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.642979 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m\" (UniqueName: \"kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m\") pod \"c4eb895b-52a2-419d-9067-5556244daebe\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " Oct 29 23:00:07.644007 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.643982 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca\") pod \"c4eb895b-52a2-419d-9067-5556244daebe\" (UID: \"c4eb895b-52a2-419d-9067-5556244daebe\") " Oct 29 23:00:07.644852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.644566 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca" pod "c4eb895b-52a2-419d-9067-5556244daebe" (UID: "c4eb895b-52a2-419d-9067-5556244daebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:07.644852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.644813 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:07.655593 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.655567 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca" pod "c4eb895b-52a2-419d-9067-5556244daebe" (UID: "c4eb895b-52a2-419d-9067-5556244daebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:07.657519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.657494 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m" pod "c4eb895b-52a2-419d-9067-5556244daebe" (UID: "c4eb895b-52a2-419d-9067-5556244daebe"). InnerVolumeSpecName "kube-api-access-l4p9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:07.746838 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.746801 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4p9m\" (UniqueName: \"kubernetes.io/projected/c4eb895b-52a2-419d-9067-5556244daebe-kube-api-access-l4p9m\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:07.746838 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:07.746840 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4eb895b-52a2-419d-9067-5556244daebe-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.371435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.371401 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" Oct 29 23:00:08.371435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.371418 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7" event={"ID":"c4eb895b-52a2-419d-9067-5556244daebe","Type":"ContainerDied","Data":"4850e6a1c997332d573ec08e60b0c097bbd327c4b34b166f36eb568196459369"} Oct 29 23:00:08.371932 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.371491 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4850e6a1c997332d573ec08e60b0c097bbd327c4b34b166f36eb568196459369" Oct 29 23:00:08.509274 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.509245 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6_20c8d02e-aa88-4237-bc6e-d18b28ffaeb6/payload-processing/0.log" Oct 29 23:00:08.576459 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.576419 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:08.661755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.661654 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca\") pod \"d25a87b5-aab0-4a48-9269-ab734ade47c7\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " Oct 29 23:00:08.661930 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.661800 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2\" (UniqueName: \"kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2\") pod \"d25a87b5-aab0-4a48-9269-ab734ade47c7\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " Oct 29 23:00:08.661930 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.661910 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca\") pod \"d25a87b5-aab0-4a48-9269-ab734ade47c7\" (UID: \"d25a87b5-aab0-4a48-9269-ab734ade47c7\") " Oct 29 23:00:08.663099 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.663071 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca" pod "d25a87b5-aab0-4a48-9269-ab734ade47c7" (UID: "d25a87b5-aab0-4a48-9269-ab734ade47c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:08.664788 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.664757 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca" pod "d25a87b5-aab0-4a48-9269-ab734ade47c7" (UID: "d25a87b5-aab0-4a48-9269-ab734ade47c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:08.667551 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.667520 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2" pod "d25a87b5-aab0-4a48-9269-ab734ade47c7" (UID: "d25a87b5-aab0-4a48-9269-ab734ade47c7"). InnerVolumeSpecName "kube-api-access-fl7w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:08.763699 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.763662 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7w2\" (UniqueName: \"kubernetes.io/projected/d25a87b5-aab0-4a48-9269-ab734ade47c7-kube-api-access-fl7w2\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.763699 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.763693 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.763699 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:08.763703 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a87b5-aab0-4a48-9269-ab734ade47c7-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.147889 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.147841 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b"] Oct 29 23:00:09.148985 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.148963 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb895b-52a2-419d-9067-5556244daebe" containerName="revoke-keys" Oct 29 23:00:09.149109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.148987 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb895b-52a2-419d-9067-5556244daebe" containerName="revoke-keys" Oct 29 23:00:09.149109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.149013 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d25a87b5-aab0-4a48-9269-ab734ade47c7" containerName="revoke-keys" Oct 29 23:00:09.149109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.149022 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25a87b5-aab0-4a48-9269-ab734ade47c7" containerName="revoke-keys" Oct 29 23:00:09.149265 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.149245 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb895b-52a2-419d-9067-5556244daebe" containerName="revoke-keys" Oct 29 23:00:09.149298 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.149264 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25a87b5-aab0-4a48-9269-ab734ade47c7" containerName="revoke-keys" Oct 29 23:00:09.153465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.153446 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.171775 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.171746 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b"] Oct 29 23:00:09.257650 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:09.257614 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:00:09.271789 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.271750 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdpb\" (UniqueName: \"kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.271948 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.271889 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.271948 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.271912 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.373311 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.373279 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdpb\" (UniqueName: \"kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.373784 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.373372 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.373784 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.373394 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.374183 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.374137 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.376642 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.376617 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.382247 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.382225 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" Oct 29 23:00:09.382356 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.382240 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272" event={"ID":"d25a87b5-aab0-4a48-9269-ab734ade47c7","Type":"ContainerDied","Data":"2378bb61583a0ab30d3a7aef973b5761999b283497ae0bfa65150eff9ae17769"} Oct 29 23:00:09.382356 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.382283 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2378bb61583a0ab30d3a7aef973b5761999b283497ae0bfa65150eff9ae17769" Oct 29 23:00:09.395489 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.395465 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdpb\" (UniqueName: \"kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.467934 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.467906 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:09.785697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:09.785670 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b"] Oct 29 23:00:10.391869 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:10.391823 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" event={"ID":"38ff0e93-fad1-4277-b3e2-e6492b504656","Type":"ContainerStarted","Data":"ca33d9c2928fdd2568f36bac571013c2cf906c8aa902cf0706ca5e065ebf4671"} Oct 29 23:00:10.392315 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:10.391894 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" event={"ID":"38ff0e93-fad1-4277-b3e2-e6492b504656","Type":"ContainerStarted","Data":"746b3b7bafb5c7ac86ed618037857e528f620d7610b803eb1c235ae094e66899"} Oct 29 23:00:10.434447 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:10.434402 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" podStartSLOduration=1.434386678 podStartE2EDuration="1.434386678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:10.433943141 +0000 UTC m=+2725.749337221" watchObservedRunningTime="2026-10-29 23:00:10.434386678 +0000 UTC m=+2725.749780760" Oct 29 23:00:11.400864 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.400829 5073 generic.go:356] "Generic (PLEG): container finished" podID="38ff0e93-fad1-4277-b3e2-e6492b504656" containerID="ca33d9c2928fdd2568f36bac571013c2cf906c8aa902cf0706ca5e065ebf4671" exitCode=0 Oct 29 23:00:11.401310 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.400918 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" event={"ID":"38ff0e93-fad1-4277-b3e2-e6492b504656","Type":"ContainerDied","Data":"ca33d9c2928fdd2568f36bac571013c2cf906c8aa902cf0706ca5e065ebf4671"} Oct 29 23:00:11.609743 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.609698 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs"] Oct 29 23:00:11.614310 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.614288 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.631185 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.631121 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e803b6-openshift-default-dockercfg-q4w9g\"" Oct 29 23:00:11.642667 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.642642 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs"] Oct 29 23:00:11.707796 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.707770 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.707936 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.707805 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.707936 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.707897 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.708006 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.707955 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.708116 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.708094 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.708249 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.708235 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.708292 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.708279 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.708379 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.708364 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.708425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.708395 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvp7\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809476 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809442 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809499 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809536 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809560 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvp7\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809619 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809639 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809956 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809675 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809956 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809727 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.809956 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.809855 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.810125 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.810051 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.810125 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.810108 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.810267 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.810186 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.810462 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.810403 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.810808 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.810783 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.812221 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.812201 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.812477 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.812461 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.833530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.833494 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.833649 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.833592 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvp7\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:11.929565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:11.929533 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:12.250264 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.250234 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs"] Oct 29 23:00:12.250663 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:12.250640 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f8dde8_ebda_49cd_b14a_825bea2f48cd.slice/crio-d8e76ce59109c204a9f7d0c387b10e5ae212bac0c3d0f225d8f96978efab3c3e WatchSource:0}: Error finding container d8e76ce59109c204a9f7d0c387b10e5ae212bac0c3d0f225d8f96978efab3c3e: Status 404 returned error can't find the container with id d8e76ce59109c204a9f7d0c387b10e5ae212bac0c3d0f225d8f96978efab3c3e Oct 29 23:00:12.254410 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.254370 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:12.254506 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.254482 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:12.254543 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.254525 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:12.412132 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.412019 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" event={"ID":"c1f8dde8-ebda-49cd-b14a-825bea2f48cd","Type":"ContainerStarted","Data":"6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13"} Oct 29 23:00:12.412132 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.412092 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" event={"ID":"c1f8dde8-ebda-49cd-b14a-825bea2f48cd","Type":"ContainerStarted","Data":"d8e76ce59109c204a9f7d0c387b10e5ae212bac0c3d0f225d8f96978efab3c3e"} Oct 29 23:00:12.463853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.463782 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podStartSLOduration=1.463761798 podStartE2EDuration="1.463761798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:12.462418885 +0000 UTC m=+2727.777812965" watchObservedRunningTime="2026-10-29 23:00:12.463761798 +0000 UTC m=+2727.779155878" Oct 29 23:00:12.605643 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.605604 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:12.723764 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.723729 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca\") pod \"38ff0e93-fad1-4277-b3e2-e6492b504656\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " Oct 29 23:00:12.723945 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.723832 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb\" (UniqueName: \"kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb\") pod \"38ff0e93-fad1-4277-b3e2-e6492b504656\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " Oct 29 23:00:12.723945 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.723852 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca\") pod \"38ff0e93-fad1-4277-b3e2-e6492b504656\" (UID: \"38ff0e93-fad1-4277-b3e2-e6492b504656\") " Oct 29 23:00:12.724217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.724192 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca" pod "38ff0e93-fad1-4277-b3e2-e6492b504656" (UID: "38ff0e93-fad1-4277-b3e2-e6492b504656"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:12.724441 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.724423 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:12.726214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.726190 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca" pod "38ff0e93-fad1-4277-b3e2-e6492b504656" (UID: "38ff0e93-fad1-4277-b3e2-e6492b504656"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:12.726289 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.726268 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb" pod "38ff0e93-fad1-4277-b3e2-e6492b504656" (UID: "38ff0e93-fad1-4277-b3e2-e6492b504656"). InnerVolumeSpecName "kube-api-access-2bdpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:12.826105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.826018 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bdpb\" (UniqueName: \"kubernetes.io/projected/38ff0e93-fad1-4277-b3e2-e6492b504656-kube-api-access-2bdpb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:12.826105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.826053 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ff0e93-fad1-4277-b3e2-e6492b504656-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:12.930000 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:12.929971 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:13.441243 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:13.441213 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" Oct 29 23:00:13.441631 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:13.441219 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b" event={"ID":"38ff0e93-fad1-4277-b3e2-e6492b504656","Type":"ContainerDied","Data":"746b3b7bafb5c7ac86ed618037857e528f620d7610b803eb1c235ae094e66899"} Oct 29 23:00:13.441631 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:13.441334 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746b3b7bafb5c7ac86ed618037857e528f620d7610b803eb1c235ae094e66899" Oct 29 23:00:13.935506 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:13.935461 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:00:13.935735 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:13.935560 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:00:14.838217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:14.834894 5073 scope.go:122] "RemoveContainer" containerID="eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69" Oct 29 23:00:14.933273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:14.932286 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:00:14.933273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:14.932348 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:00:15.340748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.335869 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97"] Oct 29 23:00:15.340748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.337345 5073 cpu_manager.go:414] "containerMap: removing container" podUID="38ff0e93-fad1-4277-b3e2-e6492b504656" containerName="revoke-keys" Oct 29 23:00:15.340748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.337371 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ff0e93-fad1-4277-b3e2-e6492b504656" containerName="revoke-keys" Oct 29 23:00:15.340748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.337620 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ff0e93-fad1-4277-b3e2-e6492b504656" containerName="revoke-keys" Oct 29 23:00:15.344685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.344653 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.375178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.371281 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:15.386173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.377638 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97"] Oct 29 23:00:15.594129 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.594035 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerStarted","Data":"05b66f58e95e1e368188285b5eecd2bed89bfe47e3f2c25005701d474d9574d6"} Oct 29 23:00:15.644985 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.644917 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.645191 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.645079 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.645191 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.645099 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vbg\" (UniqueName: \"kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.746589 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.746536 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.746778 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.746727 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.746778 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.746753 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vbg\" (UniqueName: \"kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.748415 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.747476 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.749725 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.749701 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.769767 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.769732 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vbg\" (UniqueName: \"kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg\") pod \"maas-api-revoke-sub-611836be-4hg97\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.836220 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.836127 5073 scope.go:122] "RemoveContainer" containerID="782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e" Oct 29 23:00:15.894230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.894202 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:15.940194 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.931552 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:00:15.940194 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:15.931627 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:00:16.274860 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.274732 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97"] Oct 29 23:00:16.278617 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:16.278586 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10961fa_e426_4173_aaa7_69a0444e3489.slice/crio-5032bae9732511334206d74ec40dec27bb09e12980645ce637ce34d46ef6e275 WatchSource:0}: Error finding container 5032bae9732511334206d74ec40dec27bb09e12980645ce637ce34d46ef6e275: Status 404 returned error can't find the container with id 5032bae9732511334206d74ec40dec27bb09e12980645ce637ce34d46ef6e275 Oct 29 23:00:16.603093 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.603059 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" event={"ID":"e10961fa-e426-4173-aaa7-69a0444e3489","Type":"ContainerStarted","Data":"5032bae9732511334206d74ec40dec27bb09e12980645ce637ce34d46ef6e275"} Oct 29 23:00:16.605075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.605046 5073 generic.go:356] "Generic (PLEG): container finished" podID="261794cc-16dc-4dea-8182-0ba53b471a80" containerID="05b66f58e95e1e368188285b5eecd2bed89bfe47e3f2c25005701d474d9574d6" exitCode=60 Oct 29 23:00:16.605199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.605122 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerDied","Data":"05b66f58e95e1e368188285b5eecd2bed89bfe47e3f2c25005701d474d9574d6"} Oct 29 23:00:16.605199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.605197 5073 scope.go:122] "RemoveContainer" containerID="eb4299708f9a544fea678eab1b11443087975f0ce5d5367814b14a182c7d2d69" Oct 29 23:00:16.605606 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.605585 5073 scope.go:122] "RemoveContainer" containerID="05b66f58e95e1e368188285b5eecd2bed89bfe47e3f2c25005701d474d9574d6" Oct 29 23:00:16.606010 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:16.605983 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-q8rm5_odh-ai-gateway-infra(261794cc-16dc-4dea-8182-0ba53b471a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" Oct 29 23:00:16.607219 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.607197 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerStarted","Data":"d64316bca1105de056b7e3f0e22e06e8b2e7f265b25ec4fb06595c9cbb24aefd"} Oct 29 23:00:16.653643 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.653607 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5"] Oct 29 23:00:16.873316 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.873296 5073 scope.go:122] "RemoveContainer" containerID="6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2" Oct 29 23:00:16.873519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.873497 5073 scope.go:122] "RemoveContainer" containerID="b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f" Oct 29 23:00:16.874667 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.874638 5073 scope.go:122] "RemoveContainer" containerID="295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69" Oct 29 23:00:16.943170 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.942452 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:00:16.943170 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:16.942509 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:00:17.650376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.649565 5073 generic.go:356] "Generic (PLEG): container finished" podID="156fdc41-dcc8-4740-843f-bbe96939f631" containerID="d64316bca1105de056b7e3f0e22e06e8b2e7f265b25ec4fb06595c9cbb24aefd" exitCode=60 Oct 29 23:00:17.650376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.649643 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerDied","Data":"d64316bca1105de056b7e3f0e22e06e8b2e7f265b25ec4fb06595c9cbb24aefd"} Oct 29 23:00:17.650376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.649692 5073 scope.go:122] "RemoveContainer" containerID="782ffdc1b78529fb476406a71449932df6825f836eb69489f4d9639136f7fd2e" Oct 29 23:00:17.650551 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.650458 5073 scope.go:122] "RemoveContainer" containerID="d64316bca1105de056b7e3f0e22e06e8b2e7f265b25ec4fb06595c9cbb24aefd" Oct 29 23:00:17.650930 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:17.650842 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw_odh-ai-gateway-infra(156fdc41-dcc8-4740-843f-bbe96939f631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" Oct 29 23:00:17.652685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.652657 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerStarted","Data":"40fce529c55105579654e3b140f7a22c7f72720b10fe72108627d1ad87145420"} Oct 29 23:00:17.654650 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.654626 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" event={"ID":"e10961fa-e426-4173-aaa7-69a0444e3489","Type":"ContainerStarted","Data":"d8b0e975f858fedec0b44c15a31668d10aac1b1ea5542326ca5704b489e4f853"} Oct 29 23:00:17.656635 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.656566 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerStarted","Data":"302fcaceabbebd15053c5184820540dca628a0fa1590e86ee68b1e6ce9be6c6f"} Oct 29 23:00:17.658924 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.658897 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerStarted","Data":"addf193c5e87bc77a53946639f26feb9a71d30c5a77ac50539fcef0de2063d65"} Oct 29 23:00:17.754317 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.754197 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw"] Oct 29 23:00:17.958673 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.958614 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:00:17.961330 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:17.961295 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:00:18.105772 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.105739 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:18.263696 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.263222 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca\") pod \"261794cc-16dc-4dea-8182-0ba53b471a80\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " Oct 29 23:00:18.263696 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.263374 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8\" (UniqueName: \"kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8\") pod \"261794cc-16dc-4dea-8182-0ba53b471a80\" (UID: \"261794cc-16dc-4dea-8182-0ba53b471a80\") " Oct 29 23:00:18.268527 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.268484 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca" pod "261794cc-16dc-4dea-8182-0ba53b471a80" (UID: "261794cc-16dc-4dea-8182-0ba53b471a80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.269121 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.269099 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261794cc-16dc-4dea-8182-0ba53b471a80-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.290212 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.289363 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8" pod "261794cc-16dc-4dea-8182-0ba53b471a80" (UID: "261794cc-16dc-4dea-8182-0ba53b471a80"). InnerVolumeSpecName "kube-api-access-5tqz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.370848 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.370814 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqz8\" (UniqueName: \"kubernetes.io/projected/261794cc-16dc-4dea-8182-0ba53b471a80-kube-api-access-5tqz8\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.671767 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.671667 5073 generic.go:356] "Generic (PLEG): container finished" podID="e10961fa-e426-4173-aaa7-69a0444e3489" containerID="d8b0e975f858fedec0b44c15a31668d10aac1b1ea5542326ca5704b489e4f853" exitCode=0 Oct 29 23:00:18.671931 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.671759 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" event={"ID":"e10961fa-e426-4173-aaa7-69a0444e3489","Type":"ContainerDied","Data":"d8b0e975f858fedec0b44c15a31668d10aac1b1ea5542326ca5704b489e4f853"} Oct 29 23:00:18.674562 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.674529 5073 generic.go:356] "Generic (PLEG): container finished" podID="81979462-3286-41c9-8c4e-24e619e94c36" containerID="302fcaceabbebd15053c5184820540dca628a0fa1590e86ee68b1e6ce9be6c6f" exitCode=60 Oct 29 23:00:18.674670 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.674598 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerDied","Data":"302fcaceabbebd15053c5184820540dca628a0fa1590e86ee68b1e6ce9be6c6f"} Oct 29 23:00:18.674670 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.674655 5073 scope.go:122] "RemoveContainer" containerID="295a046396b38777d3162721beadaf90a448b843fa63304f59517ac3c9c55f69" Oct 29 23:00:18.675028 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.675001 5073 scope.go:122] "RemoveContainer" containerID="302fcaceabbebd15053c5184820540dca628a0fa1590e86ee68b1e6ce9be6c6f" Oct 29 23:00:18.675493 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:18.675463 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q_odh-ai-gateway-infra(81979462-3286-41c9-8c4e-24e619e94c36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" podUID="81979462-3286-41c9-8c4e-24e619e94c36" Oct 29 23:00:18.677302 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.677280 5073 generic.go:356] "Generic (PLEG): container finished" podID="942842bf-be49-40ae-ae97-674faaf23a6e" containerID="addf193c5e87bc77a53946639f26feb9a71d30c5a77ac50539fcef0de2063d65" exitCode=60 Oct 29 23:00:18.677406 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.677367 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerDied","Data":"addf193c5e87bc77a53946639f26feb9a71d30c5a77ac50539fcef0de2063d65"} Oct 29 23:00:18.677810 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.677789 5073 scope.go:122] "RemoveContainer" containerID="addf193c5e87bc77a53946639f26feb9a71d30c5a77ac50539fcef0de2063d65" Oct 29 23:00:18.678105 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:18.678079 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj_odh-ai-gateway-infra(942842bf-be49-40ae-ae97-674faaf23a6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" Oct 29 23:00:18.678844 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.678827 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" Oct 29 23:00:18.678941 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.678848 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5" event={"ID":"261794cc-16dc-4dea-8182-0ba53b471a80","Type":"ContainerDied","Data":"140617112f5ae55dc08a3f7e7a04895d5f4352c8bf586ceb50cb15446d99bec9"} Oct 29 23:00:18.687865 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.687832 5073 generic.go:356] "Generic (PLEG): container finished" podID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerID="40fce529c55105579654e3b140f7a22c7f72720b10fe72108627d1ad87145420" exitCode=60 Oct 29 23:00:18.687986 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.687905 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerDied","Data":"40fce529c55105579654e3b140f7a22c7f72720b10fe72108627d1ad87145420"} Oct 29 23:00:18.688492 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.688277 5073 scope.go:122] "RemoveContainer" containerID="40fce529c55105579654e3b140f7a22c7f72720b10fe72108627d1ad87145420" Oct 29 23:00:18.688697 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:18.688678 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr_odh-ai-gateway-infra(87e7e415-e2ea-4cdd-adf5-784df46f6589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" Oct 29 23:00:18.697978 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.697894 5073 scope.go:122] "RemoveContainer" containerID="b32801fb3b704ac19e6d79391b34e34028dc546998d929ecef27e8d8d76e4b2f" Oct 29 23:00:18.718437 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.718417 5073 scope.go:122] "RemoveContainer" containerID="05b66f58e95e1e368188285b5eecd2bed89bfe47e3f2c25005701d474d9574d6" Oct 29 23:00:18.742480 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.742457 5073 scope.go:122] "RemoveContainer" containerID="6e7ba8b466c458bcab3d3f4fd49b6559921f4c8e2a7242062d8878837f1dc7a2" Oct 29 23:00:18.788616 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.788587 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5"] Oct 29 23:00:18.795681 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.795351 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-q8rm5"] Oct 29 23:00:18.804185 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.804120 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr"] Oct 29 23:00:18.830892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.830866 5073 scope.go:122] "RemoveContainer" containerID="13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c" Oct 29 23:00:18.849205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.845808 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" path="/var/lib/kubelet/pods/261794cc-16dc-4dea-8182-0ba53b471a80/volumes" Oct 29 23:00:18.982082 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.982040 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:18.984132 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:18.984110 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:19.068364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.068324 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q"] Oct 29 23:00:19.082340 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.082312 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj"] Oct 29 23:00:19.082731 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.082709 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca\") pod \"156fdc41-dcc8-4740-843f-bbe96939f631\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " Oct 29 23:00:19.082804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.082786 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg\" (UniqueName: \"kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg\") pod \"e10961fa-e426-4173-aaa7-69a0444e3489\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " Oct 29 23:00:19.082906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.082890 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca\") pod \"e10961fa-e426-4173-aaa7-69a0444e3489\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " Oct 29 23:00:19.083020 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.082943 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca\") pod \"e10961fa-e426-4173-aaa7-69a0444e3489\" (UID: \"e10961fa-e426-4173-aaa7-69a0444e3489\") " Oct 29 23:00:19.083020 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.082982 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd\" (UniqueName: \"kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd\") pod \"156fdc41-dcc8-4740-843f-bbe96939f631\" (UID: \"156fdc41-dcc8-4740-843f-bbe96939f631\") " Oct 29 23:00:19.083137 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.083116 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca" pod "156fdc41-dcc8-4740-843f-bbe96939f631" (UID: "156fdc41-dcc8-4740-843f-bbe96939f631"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:19.083657 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.083633 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca" pod "e10961fa-e426-4173-aaa7-69a0444e3489" (UID: "e10961fa-e426-4173-aaa7-69a0444e3489"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:19.083984 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.083737 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fdc41-dcc8-4740-843f-bbe96939f631-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.085970 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.085943 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg" pod "e10961fa-e426-4173-aaa7-69a0444e3489" (UID: "e10961fa-e426-4173-aaa7-69a0444e3489"). InnerVolumeSpecName "kube-api-access-d2vbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:19.086140 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.086120 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca" pod "e10961fa-e426-4173-aaa7-69a0444e3489" (UID: "e10961fa-e426-4173-aaa7-69a0444e3489"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:19.086451 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.086426 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd" pod "156fdc41-dcc8-4740-843f-bbe96939f631" (UID: "156fdc41-dcc8-4740-843f-bbe96939f631"). InnerVolumeSpecName "kube-api-access-hd5sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:19.174307 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.174275 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:19.175207 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.175124 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:19.176269 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.176249 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:00:19.189109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.189026 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vbg\" (UniqueName: \"kubernetes.io/projected/e10961fa-e426-4173-aaa7-69a0444e3489-kube-api-access-d2vbg\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.189109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.189064 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.189109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.189078 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10961fa-e426-4173-aaa7-69a0444e3489-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.189109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.189092 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5sd\" (UniqueName: \"kubernetes.io/projected/156fdc41-dcc8-4740-843f-bbe96939f631-kube-api-access-hd5sd\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.702757 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.702725 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerStarted","Data":"e5fff38df91d2c78df34628822562b475ed25f5e08c9b9a846c13099dac1e83b"} Oct 29 23:00:19.708193 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.708133 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" event={"ID":"156fdc41-dcc8-4740-843f-bbe96939f631","Type":"ContainerDied","Data":"1926e9cf5294e71ab44cf23ecfbb05b812e13f89b86dea82929d7733d9c27c7b"} Oct 29 23:00:19.708287 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.708237 5073 scope.go:122] "RemoveContainer" containerID="d64316bca1105de056b7e3f0e22e06e8b2e7f265b25ec4fb06595c9cbb24aefd" Oct 29 23:00:19.709344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.709319 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw" Oct 29 23:00:19.715349 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.715323 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" Oct 29 23:00:19.716853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.716695 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97" event={"ID":"e10961fa-e426-4173-aaa7-69a0444e3489","Type":"ContainerDied","Data":"5032bae9732511334206d74ec40dec27bb09e12980645ce637ce34d46ef6e275"} Oct 29 23:00:19.716853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.716749 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5032bae9732511334206d74ec40dec27bb09e12980645ce637ce34d46ef6e275" Oct 29 23:00:19.924540 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.924471 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw"] Oct 29 23:00:19.932529 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:19.932491 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8f1627-29888580-2zcfw"] Oct 29 23:00:20.022161 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.022131 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:20.026229 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.026209 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:20.029958 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.029938 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.117346 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk\" (UniqueName: \"kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk\") pod \"942842bf-be49-40ae-ae97-674faaf23a6e\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.117436 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp\" (UniqueName: \"kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp\") pod \"87e7e415-e2ea-4cdd-adf5-784df46f6589\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.117638 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw\" (UniqueName: \"kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw\") pod \"81979462-3286-41c9-8c4e-24e619e94c36\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.117736 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca\") pod \"81979462-3286-41c9-8c4e-24e619e94c36\" (UID: \"81979462-3286-41c9-8c4e-24e619e94c36\") " Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.117782 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca\") pod \"942842bf-be49-40ae-ae97-674faaf23a6e\" (UID: \"942842bf-be49-40ae-ae97-674faaf23a6e\") " Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.117855 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca\") pod \"87e7e415-e2ea-4cdd-adf5-784df46f6589\" (UID: \"87e7e415-e2ea-4cdd-adf5-784df46f6589\") " Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.119082 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca" pod "87e7e415-e2ea-4cdd-adf5-784df46f6589" (UID: "87e7e415-e2ea-4cdd-adf5-784df46f6589"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.119473 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca" pod "81979462-3286-41c9-8c4e-24e619e94c36" (UID: "81979462-3286-41c9-8c4e-24e619e94c36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.120197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.120099 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca" pod "942842bf-be49-40ae-ae97-674faaf23a6e" (UID: "942842bf-be49-40ae-ae97-674faaf23a6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.122569 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.122541 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw" pod "81979462-3286-41c9-8c4e-24e619e94c36" (UID: "81979462-3286-41c9-8c4e-24e619e94c36"). InnerVolumeSpecName "kube-api-access-vjsrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.123638 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.123610 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp" pod "87e7e415-e2ea-4cdd-adf5-784df46f6589" (UID: "87e7e415-e2ea-4cdd-adf5-784df46f6589"). InnerVolumeSpecName "kube-api-access-4fwwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.124410 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.124387 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk" pod "942842bf-be49-40ae-ae97-674faaf23a6e" (UID: "942842bf-be49-40ae-ae97-674faaf23a6e"). InnerVolumeSpecName "kube-api-access-x77tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.220017 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.219985 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x77tk\" (UniqueName: \"kubernetes.io/projected/942842bf-be49-40ae-ae97-674faaf23a6e-kube-api-access-x77tk\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.220017 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.220017 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwwp\" (UniqueName: \"kubernetes.io/projected/87e7e415-e2ea-4cdd-adf5-784df46f6589-kube-api-access-4fwwp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.220304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.220031 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjsrw\" (UniqueName: \"kubernetes.io/projected/81979462-3286-41c9-8c4e-24e619e94c36-kube-api-access-vjsrw\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.220304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.220044 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81979462-3286-41c9-8c4e-24e619e94c36-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.220304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.220056 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942842bf-be49-40ae-ae97-674faaf23a6e-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.220304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.220069 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e7e415-e2ea-4cdd-adf5-784df46f6589-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.723875 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.723843 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" Oct 29 23:00:20.724040 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.723870 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr" event={"ID":"87e7e415-e2ea-4cdd-adf5-784df46f6589","Type":"ContainerDied","Data":"13ddcc281a525ce6dfffdec114de9064cef7cdfcb73df06f99a7efe0a53f3c97"} Oct 29 23:00:20.724040 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.723960 5073 scope.go:122] "RemoveContainer" containerID="40fce529c55105579654e3b140f7a22c7f72720b10fe72108627d1ad87145420" Oct 29 23:00:20.725549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.725526 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" event={"ID":"81979462-3286-41c9-8c4e-24e619e94c36","Type":"ContainerDied","Data":"3bf0076682b7a0716b40843c56bc4829f6e74b0913b16d8a6923d2353eb1bdee"} Oct 29 23:00:20.725643 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.725630 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q" Oct 29 23:00:20.728228 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.728180 5073 generic.go:356] "Generic (PLEG): container finished" podID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerID="e5fff38df91d2c78df34628822562b475ed25f5e08c9b9a846c13099dac1e83b" exitCode=60 Oct 29 23:00:20.728365 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.728222 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerDied","Data":"e5fff38df91d2c78df34628822562b475ed25f5e08c9b9a846c13099dac1e83b"} Oct 29 23:00:20.728642 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.728622 5073 scope.go:122] "RemoveContainer" containerID="e5fff38df91d2c78df34628822562b475ed25f5e08c9b9a846c13099dac1e83b" Oct 29 23:00:20.728954 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:20.728929 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n_odh-ai-gateway-infra(cdbe33e6-8de8-447e-aef4-d52ca589bc46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" Oct 29 23:00:20.729819 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.729778 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" event={"ID":"942842bf-be49-40ae-ae97-674faaf23a6e","Type":"ContainerDied","Data":"c4774f9cb40cf81fe8b738067ef6f6b9a7ccd49d439d713a5711970e70e5f40c"} Oct 29 23:00:20.729819 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.729807 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj" Oct 29 23:00:20.737515 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.737499 5073 scope.go:122] "RemoveContainer" containerID="302fcaceabbebd15053c5184820540dca628a0fa1590e86ee68b1e6ce9be6c6f" Oct 29 23:00:20.747573 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.747553 5073 scope.go:122] "RemoveContainer" containerID="13b710a8f74f1b046a7539dbbe7dc449d27f61e4532c35c2290594c4bfaafb8c" Oct 29 23:00:20.761109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.761090 5073 scope.go:122] "RemoveContainer" containerID="addf193c5e87bc77a53946639f26feb9a71d30c5a77ac50539fcef0de2063d65" Oct 29 23:00:20.782302 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.782280 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr"] Oct 29 23:00:20.787750 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.787724 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888580-77jkr"] Oct 29 23:00:20.839999 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.839977 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" path="/var/lib/kubelet/pods/156fdc41-dcc8-4740-843f-bbe96939f631/volumes" Oct 29 23:00:20.840434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.840419 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" path="/var/lib/kubelet/pods/87e7e415-e2ea-4cdd-adf5-784df46f6589/volumes" Oct 29 23:00:20.868046 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.868026 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q"] Oct 29 23:00:20.873892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.873862 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4a0287-29888580-sfp9q"] Oct 29 23:00:20.930093 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.930069 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj"] Oct 29 23:00:20.934303 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.934280 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-afdbe8-29888580-4h5hj"] Oct 29 23:00:20.974357 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:20.974303 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n"] Oct 29 23:00:21.933698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:21.933671 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:22.043139 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.043056 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl\" (UniqueName: \"kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl\") pod \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " Oct 29 23:00:22.043324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.043289 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca\") pod \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\" (UID: \"cdbe33e6-8de8-447e-aef4-d52ca589bc46\") " Oct 29 23:00:22.043728 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.043701 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca" pod "cdbe33e6-8de8-447e-aef4-d52ca589bc46" (UID: "cdbe33e6-8de8-447e-aef4-d52ca589bc46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:22.043903 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.043888 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdbe33e6-8de8-447e-aef4-d52ca589bc46-maas-api-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.045455 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.045430 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl" pod "cdbe33e6-8de8-447e-aef4-d52ca589bc46" (UID: "cdbe33e6-8de8-447e-aef4-d52ca589bc46"). InnerVolumeSpecName "kube-api-access-6nrgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:22.145026 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.144997 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrgl\" (UniqueName: \"kubernetes.io/projected/cdbe33e6-8de8-447e-aef4-d52ca589bc46-kube-api-access-6nrgl\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.753654 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.753614 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" event={"ID":"cdbe33e6-8de8-447e-aef4-d52ca589bc46","Type":"ContainerDied","Data":"8e13c803e3f3f7a696ce89152d5be5f7d321cc86e5d98ccfa49c8840508bedec"} Oct 29 23:00:22.753654 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.753641 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n" Oct 29 23:00:22.753899 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.753686 5073 scope.go:122] "RemoveContainer" containerID="e5fff38df91d2c78df34628822562b475ed25f5e08c9b9a846c13099dac1e83b" Oct 29 23:00:22.815622 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.815595 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n"] Oct 29 23:00:22.828901 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.828877 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14b0c3-29888580-ns49n"] Oct 29 23:00:22.847636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.847602 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81979462-3286-41c9-8c4e-24e619e94c36" path="/var/lib/kubelet/pods/81979462-3286-41c9-8c4e-24e619e94c36/volumes" Oct 29 23:00:22.848539 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.848511 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" path="/var/lib/kubelet/pods/942842bf-be49-40ae-ae97-674faaf23a6e/volumes" Oct 29 23:00:22.849485 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:22.849459 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" path="/var/lib/kubelet/pods/cdbe33e6-8de8-447e-aef4-d52ca589bc46/volumes" Oct 29 23:00:23.174771 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.174674 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp"] Oct 29 23:00:23.176236 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.174916 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" podUID="0891be57-bb2a-4114-9852-bfb2b82c7a31" containerName="maas-api" containerID="cri-o://ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9" gracePeriod=30 Oct 29 23:00:23.540783 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.540750 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 23:00:23.666653 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.666624 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs\") pod \"0891be57-bb2a-4114-9852-bfb2b82c7a31\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " Oct 29 23:00:23.666823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.666740 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l\" (UniqueName: \"kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l\") pod \"0891be57-bb2a-4114-9852-bfb2b82c7a31\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " Oct 29 23:00:23.666885 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.666863 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls\") pod \"0891be57-bb2a-4114-9852-bfb2b82c7a31\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " Oct 29 23:00:23.666983 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.666968 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca\") pod \"0891be57-bb2a-4114-9852-bfb2b82c7a31\" (UID: \"0891be57-bb2a-4114-9852-bfb2b82c7a31\") " Oct 29 23:00:23.667693 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.667659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca" pod "0891be57-bb2a-4114-9852-bfb2b82c7a31" (UID: "0891be57-bb2a-4114-9852-bfb2b82c7a31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:23.667922 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.667902 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0891be57-bb2a-4114-9852-bfb2b82c7a31-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.668969 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.668944 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs" pod "0891be57-bb2a-4114-9852-bfb2b82c7a31" (UID: "0891be57-bb2a-4114-9852-bfb2b82c7a31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:23.669185 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.669139 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls" pod "0891be57-bb2a-4114-9852-bfb2b82c7a31" (UID: "0891be57-bb2a-4114-9852-bfb2b82c7a31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:23.669284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.669218 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l" pod "0891be57-bb2a-4114-9852-bfb2b82c7a31" (UID: "0891be57-bb2a-4114-9852-bfb2b82c7a31"). InnerVolumeSpecName "kube-api-access-jsp8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:23.783619 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.783529 5073 generic.go:356] "Generic (PLEG): container finished" podID="0891be57-bb2a-4114-9852-bfb2b82c7a31" containerID="ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9" exitCode=0 Oct 29 23:00:23.784097 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.784069 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" event={"ID":"0891be57-bb2a-4114-9852-bfb2b82c7a31","Type":"ContainerDied","Data":"ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9"} Oct 29 23:00:23.784301 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.784280 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" event={"ID":"0891be57-bb2a-4114-9852-bfb2b82c7a31","Type":"ContainerDied","Data":"6012e813fb460fabf2fc4dbd8527b375ea64c66937c62e8df59bb2819d6aadfd"} Oct 29 23:00:23.784419 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.784406 5073 scope.go:122] "RemoveContainer" containerID="ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9" Oct 29 23:00:23.784654 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.784638 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp" Oct 29 23:00:23.798261 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.795014 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.798261 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.795048 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsp8l\" (UniqueName: \"kubernetes.io/projected/0891be57-bb2a-4114-9852-bfb2b82c7a31-kube-api-access-jsp8l\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.798261 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.795063 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0891be57-bb2a-4114-9852-bfb2b82c7a31-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.839196 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.834774 5073 scope.go:122] "RemoveContainer" containerID="ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9" Oct 29 23:00:23.840253 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:23.839888 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9\": container with ID starting with ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9 not found: ID does not exist" containerID="ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9" Oct 29 23:00:23.840253 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.839931 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9"} err="failed to get container status \"ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9\": rpc error: code = NotFound desc = could not find container \"ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9\": container with ID starting with ecfec4bf1492af8753988607415daf17c83c9bda6db8ee8b4237acae4681a2a9 not found: ID does not exist" Oct 29 23:00:23.861300 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.861144 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp"] Oct 29 23:00:23.868279 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:23.868251 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6f83c-55d94bb86d-685lp"] Oct 29 23:00:24.178786 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:24.178561 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b"] Oct 29 23:00:24.184090 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:24.184062 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6f83c-2wk7b"] Oct 29 23:00:24.840046 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:24.840016 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0891be57-bb2a-4114-9852-bfb2b82c7a31" path="/var/lib/kubelet/pods/0891be57-bb2a-4114-9852-bfb2b82c7a31/volumes" Oct 29 23:00:24.840506 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:24.840490 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ff0e93-fad1-4277-b3e2-e6492b504656" path="/var/lib/kubelet/pods/38ff0e93-fad1-4277-b3e2-e6492b504656/volumes" Oct 29 23:00:29.504217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:29.504184 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb"] Oct 29 23:00:35.244535 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:35.244259 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp"] Oct 29 23:00:35.720895 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:35.720860 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq"] Oct 29 23:00:39.183749 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:39.183716 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct"] Oct 29 23:00:39.267986 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:39.267950 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:00:44.527602 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.527557 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" podUID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" containerName="main" containerID="cri-o://7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e" gracePeriod=45 Oct 29 23:00:44.554896 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.554865 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" podUID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": dial tcp 10.132.0.165:8000: connect: connection refused" Oct 29 23:00:44.827689 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.827664 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 23:00:44.923273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923237 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm\") pod \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " Oct 29 23:00:44.923453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923342 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home\") pod \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " Oct 29 23:00:44.923453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923378 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache\") pod \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " Oct 29 23:00:44.923453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923425 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs\") pod \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " Oct 29 23:00:44.923645 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923539 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84\" (UniqueName: \"kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84\") pod \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " Oct 29 23:00:44.923711 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923640 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home" pod "20bbe73c-4a61-477c-a505-91bdf13d1dd3" (UID: "20bbe73c-4a61-477c-a505-91bdf13d1dd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:44.923772 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923757 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir\") pod \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\" (UID: \"20bbe73c-4a61-477c-a505-91bdf13d1dd3\") " Oct 29 23:00:44.923829 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.923788 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache" pod "20bbe73c-4a61-477c-a505-91bdf13d1dd3" (UID: "20bbe73c-4a61-477c-a505-91bdf13d1dd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:44.924353 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.924329 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:44.924535 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.924518 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:44.924833 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.924805 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir" pod "20bbe73c-4a61-477c-a505-91bdf13d1dd3" (UID: "20bbe73c-4a61-477c-a505-91bdf13d1dd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:44.927686 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.927660 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84" pod "20bbe73c-4a61-477c-a505-91bdf13d1dd3" (UID: "20bbe73c-4a61-477c-a505-91bdf13d1dd3"). InnerVolumeSpecName "kube-api-access-2gw84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:44.927801 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.927688 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm" pod "20bbe73c-4a61-477c-a505-91bdf13d1dd3" (UID: "20bbe73c-4a61-477c-a505-91bdf13d1dd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:44.928165 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:44.928126 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs" pod "20bbe73c-4a61-477c-a505-91bdf13d1dd3" (UID: "20bbe73c-4a61-477c-a505-91bdf13d1dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:45.017498 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.017469 5073 generic.go:356] "Generic (PLEG): container finished" podID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" containerID="7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e" exitCode=0 Oct 29 23:00:45.017624 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.017514 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" Oct 29 23:00:45.017624 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.017556 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" event={"ID":"20bbe73c-4a61-477c-a505-91bdf13d1dd3","Type":"ContainerDied","Data":"7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e"} Oct 29 23:00:45.017624 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.017606 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb" event={"ID":"20bbe73c-4a61-477c-a505-91bdf13d1dd3","Type":"ContainerDied","Data":"c0e1413cbe640d9094a7bc06e031685c8b3ddca240137d7b6ab5c4d1bd20b9f9"} Oct 29 23:00:45.017761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.017632 5073 scope.go:122] "RemoveContainer" containerID="7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e" Oct 29 23:00:45.025876 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.025847 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:45.025969 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.025879 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bbe73c-4a61-477c-a505-91bdf13d1dd3-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:45.025969 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.025890 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bbe73c-4a61-477c-a505-91bdf13d1dd3-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:45.025969 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.025898 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gw84\" (UniqueName: \"kubernetes.io/projected/20bbe73c-4a61-477c-a505-91bdf13d1dd3-kube-api-access-2gw84\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:45.028455 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.028435 5073 scope.go:122] "RemoveContainer" containerID="7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e" Oct 29 23:00:45.028690 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:45.028672 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e\": container with ID starting with 7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e not found: ID does not exist" containerID="7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e" Oct 29 23:00:45.028738 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.028698 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e"} err="failed to get container status \"7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e\": rpc error: code = NotFound desc = could not find container \"7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e\": container with ID starting with 7a618242aef38d972e8fa4c60edfae4c369d1f1ff5bb008054477f5dba252d2e not found: ID does not exist" Oct 29 23:00:45.094931 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.094850 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb"] Oct 29 23:00:45.098903 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:45.098876 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-329f7b/auth-test-model-329f7b-kserve-54997dc6fc-m9cqb"] Oct 29 23:00:46.839056 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:46.839022 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" path="/var/lib/kubelet/pods/20bbe73c-4a61-477c-a505-91bdf13d1dd3/volumes" Oct 29 23:00:50.283373 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.283322 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" podUID="0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" containerName="main" containerID="cri-o://44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e" gracePeriod=45 Oct 29 23:00:50.585258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.585228 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 23:00:50.665429 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665392 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm\") pod \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " Oct 29 23:00:50.665610 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665544 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache\") pod \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " Oct 29 23:00:50.665610 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665583 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir\") pod \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " Oct 29 23:00:50.665749 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665676 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z\" (UniqueName: \"kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z\") pod \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " Oct 29 23:00:50.665749 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665743 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home\") pod \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " Oct 29 23:00:50.665861 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665837 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs\") pod \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\" (UID: \"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1\") " Oct 29 23:00:50.665995 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.665971 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir" pod "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" (UID: "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.666109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.666091 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache" pod "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" (UID: "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.666205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.666100 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home" pod "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" (UID: "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.666581 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.666550 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.666662 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.666583 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.666662 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.666596 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.668743 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.668711 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs" pod "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" (UID: "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:50.669102 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.669074 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z" pod "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" (UID: "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1"). InnerVolumeSpecName "kube-api-access-7wg9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:50.669272 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.669106 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm" pod "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" (UID: "0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.743091 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.743058 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" podUID="0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" containerName="main" containerID="cri-o://b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f" gracePeriod=45 Oct 29 23:00:50.768223 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.768198 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wg9z\" (UniqueName: \"kubernetes.io/projected/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-kube-api-access-7wg9z\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.768314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.768225 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.768314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:50.768236 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.033717 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.033691 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 23:00:51.065737 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.065710 5073 generic.go:356] "Generic (PLEG): container finished" podID="0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" containerID="b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f" exitCode=0 Oct 29 23:00:51.065887 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.065752 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" Oct 29 23:00:51.065887 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.065794 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" event={"ID":"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83","Type":"ContainerDied","Data":"b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f"} Oct 29 23:00:51.065887 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.065835 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq" event={"ID":"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83","Type":"ContainerDied","Data":"320d34acf7d63871e9bc40c5b80532ed577a7f0d432cb45b20306aeee60ade5f"} Oct 29 23:00:51.065887 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.065853 5073 scope.go:122] "RemoveContainer" containerID="b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f" Oct 29 23:00:51.067609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.067587 5073 generic.go:356] "Generic (PLEG): container finished" podID="0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" containerID="44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e" exitCode=0 Oct 29 23:00:51.067708 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.067621 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" event={"ID":"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1","Type":"ContainerDied","Data":"44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e"} Oct 29 23:00:51.067708 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.067642 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" Oct 29 23:00:51.067708 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.067661 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp" event={"ID":"0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1","Type":"ContainerDied","Data":"e8a24d905f6b1de0bd1e3199f40903a6cfcbfb7d4b05e45c35743175b3513710"} Oct 29 23:00:51.076269 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.076252 5073 scope.go:122] "RemoveContainer" containerID="b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f" Oct 29 23:00:51.076511 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:51.076491 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f\": container with ID starting with b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f not found: ID does not exist" containerID="b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f" Oct 29 23:00:51.076569 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.076520 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f"} err="failed to get container status \"b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f\": rpc error: code = NotFound desc = could not find container \"b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f\": container with ID starting with b598186cf0c964f60ffcb258952b2b94b659769836af736b9792fd58dbcc069f not found: ID does not exist" Oct 29 23:00:51.076569 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.076536 5073 scope.go:122] "RemoveContainer" containerID="44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e" Oct 29 23:00:51.089571 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.089547 5073 scope.go:122] "RemoveContainer" containerID="44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e" Oct 29 23:00:51.089948 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:51.089922 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e\": container with ID starting with 44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e not found: ID does not exist" containerID="44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e" Oct 29 23:00:51.090031 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.089959 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e"} err="failed to get container status \"44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e\": rpc error: code = NotFound desc = could not find container \"44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e\": container with ID starting with 44c021811125f97d05723981df3777fa47e162ef20f6a0aa684df3c06979e82e not found: ID does not exist" Oct 29 23:00:51.123648 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.123579 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp"] Oct 29 23:00:51.127834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.127811 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a15f4/auth-test-model-9a15f4-kserve-654f88d7cc-dthbp"] Oct 29 23:00:51.173680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.173657 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm\") pod \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " Oct 29 23:00:51.173948 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.173813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache\") pod \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " Oct 29 23:00:51.173948 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.173891 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir\") pod \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " Oct 29 23:00:51.174095 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.173955 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs\" (UniqueName: \"kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs\") pod \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " Oct 29 23:00:51.174095 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174026 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home\") pod \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " Oct 29 23:00:51.174095 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174038 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache" pod "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" (UID: "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:51.174095 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174090 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj\" (UniqueName: \"kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj\") pod \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\" (UID: \"0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83\") " Oct 29 23:00:51.174347 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174222 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir" pod "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" (UID: "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:51.174410 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174344 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home" pod "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" (UID: "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:51.174688 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174643 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.174688 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174667 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.174688 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.174681 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.176752 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.176727 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs" pod "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" (UID: "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:51.176752 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.176745 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm" pod "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" (UID: "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:51.176905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.176890 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj" pod "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" (UID: "0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83"). InnerVolumeSpecName "kube-api-access-mdzxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:51.275892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.275867 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzxj\" (UniqueName: \"kubernetes.io/projected/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-kube-api-access-mdzxj\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.275892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.275891 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.276029 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.275899 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:51.414834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.414791 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq"] Oct 29 23:00:51.420911 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:51.420891 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-36b46d/auth-test-model-36b46d-kserve-6fb4f7bdb9-s2lvq"] Oct 29 23:00:52.839754 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:52.839719 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" path="/var/lib/kubelet/pods/0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83/volumes" Oct 29 23:00:52.840390 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:52.840368 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" path="/var/lib/kubelet/pods/0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1/volumes" Oct 29 23:00:54.027637 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.027602 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2"] Oct 29 23:00:54.219174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.218426 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" podUID="780ac08b-7550-49ee-8b62-141339eb9ff7" containerName="main" containerID="cri-o://f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b" gracePeriod=45 Oct 29 23:00:54.522728 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.522702 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 23:00:54.550876 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.550844 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm\") pod \"780ac08b-7550-49ee-8b62-141339eb9ff7\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " Oct 29 23:00:54.551021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.550930 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs\" (UniqueName: \"kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs\") pod \"780ac08b-7550-49ee-8b62-141339eb9ff7\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " Oct 29 23:00:54.551021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.550969 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir\") pod \"780ac08b-7550-49ee-8b62-141339eb9ff7\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " Oct 29 23:00:54.551021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551014 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache\") pod \"780ac08b-7550-49ee-8b62-141339eb9ff7\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " Oct 29 23:00:54.551251 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551047 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np\" (UniqueName: \"kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np\") pod \"780ac08b-7550-49ee-8b62-141339eb9ff7\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " Oct 29 23:00:54.551251 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551187 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home\") pod \"780ac08b-7550-49ee-8b62-141339eb9ff7\" (UID: \"780ac08b-7550-49ee-8b62-141339eb9ff7\") " Oct 29 23:00:54.551359 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551315 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir" pod "780ac08b-7550-49ee-8b62-141339eb9ff7" (UID: "780ac08b-7550-49ee-8b62-141339eb9ff7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:54.551417 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551351 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache" pod "780ac08b-7550-49ee-8b62-141339eb9ff7" (UID: "780ac08b-7550-49ee-8b62-141339eb9ff7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:54.551610 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551588 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home" pod "780ac08b-7550-49ee-8b62-141339eb9ff7" (UID: "780ac08b-7550-49ee-8b62-141339eb9ff7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:54.551869 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551849 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:54.551869 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551871 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:54.552014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.551884 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:54.553308 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.553287 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm" pod "780ac08b-7550-49ee-8b62-141339eb9ff7" (UID: "780ac08b-7550-49ee-8b62-141339eb9ff7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:54.553761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.553742 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs" pod "780ac08b-7550-49ee-8b62-141339eb9ff7" (UID: "780ac08b-7550-49ee-8b62-141339eb9ff7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:54.553806 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.553791 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np" pod "780ac08b-7550-49ee-8b62-141339eb9ff7" (UID: "780ac08b-7550-49ee-8b62-141339eb9ff7"). InnerVolumeSpecName "kube-api-access-mm4np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:54.654014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.653918 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780ac08b-7550-49ee-8b62-141339eb9ff7-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:54.654014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.653956 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4np\" (UniqueName: \"kubernetes.io/projected/780ac08b-7550-49ee-8b62-141339eb9ff7-kube-api-access-mm4np\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:54.654014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:54.653969 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780ac08b-7550-49ee-8b62-141339eb9ff7-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:00:55.105588 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.105547 5073 generic.go:356] "Generic (PLEG): container finished" podID="780ac08b-7550-49ee-8b62-141339eb9ff7" containerID="f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b" exitCode=0 Oct 29 23:00:55.106101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.105594 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" Oct 29 23:00:55.106101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.105629 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" event={"ID":"780ac08b-7550-49ee-8b62-141339eb9ff7","Type":"ContainerDied","Data":"f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b"} Oct 29 23:00:55.106101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.105686 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct" event={"ID":"780ac08b-7550-49ee-8b62-141339eb9ff7","Type":"ContainerDied","Data":"6cbb1369d32aadeed18dacddc7dec3d2949795f0763928f60a27d0e62242c372"} Oct 29 23:00:55.106101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.105711 5073 scope.go:122] "RemoveContainer" containerID="f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b" Oct 29 23:00:55.118263 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.118244 5073 scope.go:122] "RemoveContainer" containerID="f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b" Oct 29 23:00:55.118589 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:00:55.118568 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b\": container with ID starting with f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b not found: ID does not exist" containerID="f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b" Oct 29 23:00:55.118671 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.118599 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b"} err="failed to get container status \"f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b\": rpc error: code = NotFound desc = could not find container \"f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b\": container with ID starting with f54f0071d218a7d6418a394149d72b3d988f58fd36282158b389f4a76772366b not found: ID does not exist" Oct 29 23:00:55.186545 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.186517 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct"] Oct 29 23:00:55.200180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.200122 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2ae845/auth-test-model-2ae845-kserve-5c4cc4d689-cwlct"] Oct 29 23:00:55.268962 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.268915 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn"] Oct 29 23:00:55.270217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270191 5073 cpu_manager.go:414] "containerMap: removing container" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.270350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270336 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.270463 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270450 5073 cpu_manager.go:414] "containerMap: removing container" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.270547 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270535 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.270642 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270631 5073 cpu_manager.go:414] "containerMap: removing container" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:55.270718 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270708 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:55.270810 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270799 5073 cpu_manager.go:414] "containerMap: removing container" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:55.270892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270881 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:55.270980 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.270970 5073 cpu_manager.go:414] "containerMap: removing container" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.271087 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271073 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.271228 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271214 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0891be57-bb2a-4114-9852-bfb2b82c7a31" containerName="maas-api" Oct 29 23:00:55.271323 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271311 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0891be57-bb2a-4114-9852-bfb2b82c7a31" containerName="maas-api" Oct 29 23:00:55.271415 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271405 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" containerName="main" Oct 29 23:00:55.271491 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271480 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" containerName="main" Oct 29 23:00:55.271575 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271565 5073 cpu_manager.go:414] "containerMap: removing container" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:55.271656 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271645 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:55.271739 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271729 5073 cpu_manager.go:414] "containerMap: removing container" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:55.271818 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271807 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:55.271900 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271889 5073 cpu_manager.go:414] "containerMap: removing container" podUID="780ac08b-7550-49ee-8b62-141339eb9ff7" containerName="main" Oct 29 23:00:55.271972 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.271962 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780ac08b-7550-49ee-8b62-141339eb9ff7" containerName="main" Oct 29 23:00:55.272064 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272054 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.272218 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272134 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.272335 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272320 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.272415 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272404 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.272516 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272504 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" containerName="main" Oct 29 23:00:55.272592 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272582 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" containerName="main" Oct 29 23:00:55.272686 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272676 5073 cpu_manager.go:414] "containerMap: removing container" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:55.272771 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272759 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:55.272869 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272843 5073 cpu_manager.go:414] "containerMap: removing container" podUID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" containerName="main" Oct 29 23:00:55.272977 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.272963 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" containerName="main" Oct 29 23:00:55.273071 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273060 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e10961fa-e426-4173-aaa7-69a0444e3489" containerName="revoke-keys" Oct 29 23:00:55.273166 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273137 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10961fa-e426-4173-aaa7-69a0444e3489" containerName="revoke-keys" Oct 29 23:00:55.273327 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273313 5073 cpu_manager.go:414] "containerMap: removing container" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:55.273414 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273403 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:55.273520 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273509 5073 cpu_manager.go:414] "containerMap: removing container" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.273603 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273593 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.273855 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273837 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="780ac08b-7550-49ee-8b62-141339eb9ff7" containerName="main" Oct 29 23:00:55.273980 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.273966 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.274073 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274063 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.274182 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274168 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aaa2d63-47c8-49bd-a9ba-ac1dbdd84b83" containerName="main" Oct 29 23:00:55.274289 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274278 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:55.274420 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274408 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bbe73c-4a61-477c-a505-91bdf13d1dd3" containerName="main" Oct 29 23:00:55.274530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274504 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.274635 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274623 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10961fa-e426-4173-aaa7-69a0444e3489" containerName="revoke-keys" Oct 29 23:00:55.274720 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274710 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:55.274877 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274863 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:55.274984 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.274973 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:55.275074 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275063 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:55.275177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275164 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.275273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275263 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.275358 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275348 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.275435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275426 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:55.275514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275505 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:55.275605 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275595 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0891be57-bb2a-4114-9852-bfb2b82c7a31" containerName="maas-api" Oct 29 23:00:55.275687 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275677 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9ed871-b5c6-422b-b77d-7b70aa5fbfe1" containerName="main" Oct 29 23:00:55.275766 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275756 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:55.275851 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.275831 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:55.281585 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.281566 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.322766 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.322737 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn"] Oct 29 23:00:55.328377 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.328345 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3e2b2c-openshift-default-dockercfg-q5pmh\"" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.429905 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.429969 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430049 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430134 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430201 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430250 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430367 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szj7g\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430396 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.433199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.430444 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.531830 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.531797 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532004 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.531842 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532004 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.531917 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532004 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532001 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532228 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532046 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532228 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532096 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532229 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szj7g\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532259 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532305 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532502 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532356 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532724 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532701 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.532769 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.532742 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.533234 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.533179 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.533234 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.533217 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.534853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.534794 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.536888 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.536866 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.554920 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.554895 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.555580 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.555559 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szj7g\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g\") pod \"e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.598866 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.598838 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:55.938640 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.938418 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn"] Oct 29 23:00:55.940697 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:55.940666 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64aaf8d_64fe_4420_bdb5_a604c82f4305.slice/crio-ff920fc1da3737804edb4a7d0d45dee016046b690f6fd1c4afa9d4a3b617474f WatchSource:0}: Error finding container ff920fc1da3737804edb4a7d0d45dee016046b690f6fd1c4afa9d4a3b617474f: Status 404 returned error can't find the container with id ff920fc1da3737804edb4a7d0d45dee016046b690f6fd1c4afa9d4a3b617474f Oct 29 23:00:55.943935 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.943897 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:55.944030 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.944003 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:55.944083 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:55.944041 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:56.120841 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.120806 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" event={"ID":"e64aaf8d-64fe-4420-bdb5-a604c82f4305","Type":"ContainerStarted","Data":"aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea"} Oct 29 23:00:56.121286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.120863 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" event={"ID":"e64aaf8d-64fe-4420-bdb5-a604c82f4305","Type":"ContainerStarted","Data":"ff920fc1da3737804edb4a7d0d45dee016046b690f6fd1c4afa9d4a3b617474f"} Oct 29 23:00:56.180031 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.179953 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podStartSLOduration=1.179933416 podStartE2EDuration="1.179933416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:56.179145067 +0000 UTC m=+2771.494539152" watchObservedRunningTime="2026-10-29 23:00:56.179933416 +0000 UTC m=+2771.495327500" Oct 29 23:00:56.599241 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.599208 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:56.839188 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.839142 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780ac08b-7550-49ee-8b62-141339eb9ff7" path="/var/lib/kubelet/pods/780ac08b-7550-49ee-8b62-141339eb9ff7/volumes" Oct 29 23:00:56.897510 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.897439 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9"] Oct 29 23:00:56.898064 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898049 5073 cpu_manager.go:414] "containerMap: removing container" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:56.898117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898065 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156fdc41-dcc8-4740-843f-bbe96939f631" containerName="cleanup" Oct 29 23:00:56.898117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898075 5073 cpu_manager.go:414] "containerMap: removing container" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:56.898117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898080 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:56.898117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898094 5073 cpu_manager.go:414] "containerMap: removing container" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:56.898117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898099 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942842bf-be49-40ae-ae97-674faaf23a6e" containerName="cleanup" Oct 29 23:00:56.898117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898114 5073 cpu_manager.go:414] "containerMap: removing container" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898120 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898137 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898142 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbe33e6-8de8-447e-aef4-d52ca589bc46" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898174 5073 cpu_manager.go:414] "containerMap: removing container" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898179 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898274 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e7e415-e2ea-4cdd-adf5-784df46f6589" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898286 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="261794cc-16dc-4dea-8182-0ba53b471a80" containerName="cleanup" Oct 29 23:00:56.898324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.898295 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="81979462-3286-41c9-8c4e-24e619e94c36" containerName="cleanup" Oct 29 23:00:56.902008 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.901990 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:56.940712 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:56.940680 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9"] Oct 29 23:00:57.060723 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.060676 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.060946 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.060736 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.060946 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.060789 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.060946 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.060879 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.061118 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.061061 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.061556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.061131 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljw9\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.061556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.061228 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.061556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.061258 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.061556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.061288 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.162418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162326 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.162418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162374 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.162418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162412 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162504 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162615 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162674 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljw9\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162756 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162773 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162787 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162812 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162829 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163003 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.162863 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163341 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.163118 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.163452 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.163433 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.165229 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.165205 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.165469 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.165452 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.188691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.188664 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.188960 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.188938 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljw9\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.216983 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.216957 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:57.544634 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:57.544603 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca898de_4510_4f38_b7f9_bfb6a32cb466.slice/crio-95734577edbaa077d7c0a1d82900bf9948076b125593f9efba5bfe0fd24c8ace WatchSource:0}: Error finding container 95734577edbaa077d7c0a1d82900bf9948076b125593f9efba5bfe0fd24c8ace: Status 404 returned error can't find the container with id 95734577edbaa077d7c0a1d82900bf9948076b125593f9efba5bfe0fd24c8ace Oct 29 23:00:57.546679 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.546653 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9"] Oct 29 23:00:57.547532 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.547497 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:57.547588 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.547565 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:57.547621 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.547606 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:57.606365 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:57.606340 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:58.141824 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.141787 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" event={"ID":"eca898de-4510-4f38-b7f9-bfb6a32cb466","Type":"ContainerStarted","Data":"b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5"} Oct 29 23:00:58.142014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.141845 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" event={"ID":"eca898de-4510-4f38-b7f9-bfb6a32cb466","Type":"ContainerStarted","Data":"95734577edbaa077d7c0a1d82900bf9948076b125593f9efba5bfe0fd24c8ace"} Oct 29 23:00:58.142414 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.142392 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:58.143569 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.143547 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:00:58.195387 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.195343 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podStartSLOduration=2.195329927 podStartE2EDuration="2.195329927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:58.190200731 +0000 UTC m=+2773.505594811" watchObservedRunningTime="2026-10-29 23:00:58.195329927 +0000 UTC m=+2773.510724007" Oct 29 23:00:58.230176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.228522 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:00:58.233327 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.233282 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:00:58.233473 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:58.233362 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:00:59.219438 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.218567 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:00:59.219438 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.218636 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:00:59.243228 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.243192 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4"] Oct 29 23:00:59.249520 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.249488 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.296565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.296527 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8f1627-openshift-default-dockercfg-qjqk5\"" Oct 29 23:00:59.300471 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.300447 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4"] Oct 29 23:00:59.406140 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406110 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406338 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406274 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406396 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406347 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vjw\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406461 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406421 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406505 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406499 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406577 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406560 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406632 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406592 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406707 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406695 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.406748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.406725 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508356 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508439 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508474 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508593 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508640 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508705 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508784 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508838 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vjw\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.515230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.508929 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.533173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.516626 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.533173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.516915 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.533173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.517166 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.533173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.518720 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.533173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.518974 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.536033 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.536004 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.536329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.536306 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.545544 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.545520 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.552219 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.552196 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vjw\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.569076 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.569053 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:00:59.938348 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:00:59.938319 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77dbabf1_951f_42d5_b02f_363085127b26.slice/crio-09da10a33bfed7577352c7e5bc3c2f4e8874d71d4709861c29cc5df851be2078 WatchSource:0}: Error finding container 09da10a33bfed7577352c7e5bc3c2f4e8874d71d4709861c29cc5df851be2078: Status 404 returned error can't find the container with id 09da10a33bfed7577352c7e5bc3c2f4e8874d71d4709861c29cc5df851be2078 Oct 29 23:00:59.938714 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.938687 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4"] Oct 29 23:00:59.941321 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.941277 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:59.941416 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.941357 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:00:59.941416 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:00:59.941402 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:00.160351 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:00.160313 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" event={"ID":"77dbabf1-951f-42d5-b02f-363085127b26","Type":"ContainerStarted","Data":"51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad"} Oct 29 23:01:00.160530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:00.160377 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" event={"ID":"77dbabf1-951f-42d5-b02f-363085127b26","Type":"ContainerStarted","Data":"09da10a33bfed7577352c7e5bc3c2f4e8874d71d4709861c29cc5df851be2078"} Oct 29 23:01:00.226281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:00.226247 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:01:00.230241 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:00.230204 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:01:00.235507 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:00.235456 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podStartSLOduration=1.235438903 podStartE2EDuration="1.235438903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:00.230874633 +0000 UTC m=+2775.546268730" watchObservedRunningTime="2026-10-29 23:01:00.235438903 +0000 UTC m=+2775.550832985" Oct 29 23:01:00.570019 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:00.569939 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:01:01.043969 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.043931 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw"] Oct 29 23:01:01.095730 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.095699 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw"] Oct 29 23:01:01.095992 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.095812 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.133172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.118434 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 23:01:01.133172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.118657 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cvlmz\"" Oct 29 23:01:01.133172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.118708 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 23:01:01.155204 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.139287 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 23:01:01.188791 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.188490 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.188989 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.188847 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.188989 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.188930 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrgg\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.188989 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.188958 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.188989 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.188981 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.189233 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.189006 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.189446 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.189422 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.258042 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.258006 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:01:01.258666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.258279 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:01:01.290914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.290885 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.291065 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.290975 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.291065 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.291029 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.291224 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.291092 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrgg\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.291224 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.291113 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.291224 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.291128 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.291224 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.291144 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.292293 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.292268 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.293817 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.293788 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.293817 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.293809 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.294112 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.294075 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.294283 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.294266 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.361175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.358674 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.361175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.359299 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrgg\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lkrpw\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.459080 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.457427 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:01.572178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.572120 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:01:01.572292 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.572219 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 29 23:01:01.898875 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.898842 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw"] Oct 29 23:01:01.905132 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.904503 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:01.905132 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:01.904595 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:02.189384 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.189269 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" event={"ID":"78256170-0e88-45a6-84aa-f4c81d35d722","Type":"ContainerStarted","Data":"e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420"} Oct 29 23:01:02.190360 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.190323 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" event={"ID":"78256170-0e88-45a6-84aa-f4c81d35d722","Type":"ContainerStarted","Data":"fc87623e20246e15000a7f8c89068873164beca4e54f46885ba0822fe1a5009b"} Oct 29 23:01:02.190528 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.190398 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:02.230178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.218594 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:01:02.230178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.218644 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:01:02.277210 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.277129 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" podStartSLOduration=2.277114197 podStartE2EDuration="2.277114197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:02.273653859 +0000 UTC m=+2777.589047940" watchObservedRunningTime="2026-10-29 23:01:02.277114197 +0000 UTC m=+2777.592508278" Oct 29 23:01:02.570173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.569617 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:01:02.570173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:02.569680 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 29 23:01:03.222029 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:03.221989 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.226417 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:03.226382 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:01:03.252176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:03.248316 5073 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lkrpw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:03.252176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:03.248377 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:03.571230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:03.571124 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.571823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:03.571773 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 29 23:01:04.105558 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.104171 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" podUID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": context deadline exceeded" Oct 29 23:01:04.252558 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.252524 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:01:04.273530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.252810 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:01:04.345301 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.345255 5073 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lkrpw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:04.345577 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.345551 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:04.542553 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.522012 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5"] Oct 29 23:01:04.549437 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.549397 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" podUID="c789c8b7-746a-43e3-b087-0f8b291cb446" containerName="maas-api" containerID="cri-o://be6fd702782683b1faa3a1945cfdce8e47c75ace9570e280163bb9a2e25a42eb" gracePeriod=30 Oct 29 23:01:04.578206 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.571577 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:01:04.578206 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.571638 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 29 23:01:04.720339 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:04.720298 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" podUID="c789c8b7-746a-43e3-b087-0f8b291cb446" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.47:8443/health\": dial tcp 10.132.0.47:8443: connect: connection refused" Oct 29 23:01:05.362795 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.362760 5073 generic.go:356] "Generic (PLEG): container finished" podID="c789c8b7-746a-43e3-b087-0f8b291cb446" containerID="be6fd702782683b1faa3a1945cfdce8e47c75ace9570e280163bb9a2e25a42eb" exitCode=0 Oct 29 23:01:05.363297 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.363271 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" event={"ID":"c789c8b7-746a-43e3-b087-0f8b291cb446","Type":"ContainerDied","Data":"be6fd702782683b1faa3a1945cfdce8e47c75ace9570e280163bb9a2e25a42eb"} Oct 29 23:01:05.451230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.449093 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 23:01:05.474613 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.474577 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:01:05.476519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.476484 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:01:05.479213 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.479180 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:01:05.566825 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.566743 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm\" (UniqueName: \"kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm\") pod \"c789c8b7-746a-43e3-b087-0f8b291cb446\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " Oct 29 23:01:05.566966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.566937 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs\") pod \"c789c8b7-746a-43e3-b087-0f8b291cb446\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " Oct 29 23:01:05.567137 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.567120 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca\") pod \"c789c8b7-746a-43e3-b087-0f8b291cb446\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " Oct 29 23:01:05.567263 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.567249 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls\") pod \"c789c8b7-746a-43e3-b087-0f8b291cb446\" (UID: \"c789c8b7-746a-43e3-b087-0f8b291cb446\") " Oct 29 23:01:05.567774 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.567664 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca" pod "c789c8b7-746a-43e3-b087-0f8b291cb446" (UID: "c789c8b7-746a-43e3-b087-0f8b291cb446"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:05.568257 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.568225 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789c8b7-746a-43e3-b087-0f8b291cb446-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:05.569357 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.569326 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm" pod "c789c8b7-746a-43e3-b087-0f8b291cb446" (UID: "c789c8b7-746a-43e3-b087-0f8b291cb446"). InnerVolumeSpecName "kube-api-access-d7vwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:05.569481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.569461 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:01:05.569481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.569469 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs" pod "c789c8b7-746a-43e3-b087-0f8b291cb446" (UID: "c789c8b7-746a-43e3-b087-0f8b291cb446"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:05.569622 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.569509 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 29 23:01:05.569622 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.569526 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls" pod "c789c8b7-746a-43e3-b087-0f8b291cb446" (UID: "c789c8b7-746a-43e3-b087-0f8b291cb446"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:05.669710 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.669677 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:05.669710 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.669708 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vwm\" (UniqueName: \"kubernetes.io/projected/c789c8b7-746a-43e3-b087-0f8b291cb446-kube-api-access-d7vwm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:05.669710 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:05.669718 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789c8b7-746a-43e3-b087-0f8b291cb446-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:06.371609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.371582 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" Oct 29 23:01:06.371889 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.371581 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5" event={"ID":"c789c8b7-746a-43e3-b087-0f8b291cb446","Type":"ContainerDied","Data":"8373a0e3ce31baa35bb1e57cdbf0e2c6865e9a10917e92f0e2b1aabf555d0dae"} Oct 29 23:01:06.371889 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.371727 5073 scope.go:122] "RemoveContainer" containerID="be6fd702782683b1faa3a1945cfdce8e47c75ace9570e280163bb9a2e25a42eb" Oct 29 23:01:06.503257 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.503212 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5"] Oct 29 23:01:06.553328 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.553284 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40cf9a-884d5bdb7-z94d5"] Oct 29 23:01:06.571210 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.571175 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:01:06.571497 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.571469 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 29 23:01:06.841982 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:06.841950 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c789c8b7-746a-43e3-b087-0f8b291cb446" path="/var/lib/kubelet/pods/c789c8b7-746a-43e3-b087-0f8b291cb446/volumes" Oct 29 23:01:07.335201 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:07.335174 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:01:07.574104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:07.574076 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:01:07.574535 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:07.574514 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:01:07.575270 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:07.575250 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:01:09.173424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.173384 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" podUID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" containerName="main" containerID="cri-o://eebd23514edd354826d3e704d80a093528f3e9f930c0de869ff756af8f469ae6" gracePeriod=45 Oct 29 23:01:09.254615 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:01:09.254576 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:01:09.403250 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.403220 5073 generic.go:356] "Generic (PLEG): container finished" podID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" containerID="eebd23514edd354826d3e704d80a093528f3e9f930c0de869ff756af8f469ae6" exitCode=0 Oct 29 23:01:09.403438 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.403309 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" event={"ID":"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab","Type":"ContainerDied","Data":"eebd23514edd354826d3e704d80a093528f3e9f930c0de869ff756af8f469ae6"} Oct 29 23:01:09.480895 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.480865 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 23:01:09.524335 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524300 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir\") pod \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " Oct 29 23:01:09.524457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524355 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs\") pod \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " Oct 29 23:01:09.524565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524547 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home\") pod \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " Oct 29 23:01:09.524616 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524576 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir" pod "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" (UID: "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:09.524676 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524642 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm\") pod \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " Oct 29 23:01:09.524737 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524693 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache\") pod \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " Oct 29 23:01:09.524794 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524736 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4\" (UniqueName: \"kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4\") pod \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\" (UID: \"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab\") " Oct 29 23:01:09.524857 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524815 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home" pod "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" (UID: "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:09.525022 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.524999 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache" pod "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" (UID: "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:09.525615 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.525589 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:09.525722 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.525622 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:09.525722 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.525637 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:09.527001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.526973 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs" pod "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" (UID: "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:09.527001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.526980 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm" pod "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" (UID: "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:09.527124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.527064 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4" pod "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" (UID: "20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab"). InnerVolumeSpecName "kube-api-access-bkzs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:09.627103 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.627076 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:09.627103 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.627102 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:09.627301 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:09.627112 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzs4\" (UniqueName: \"kubernetes.io/projected/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab-kube-api-access-bkzs4\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:10.034136 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034099 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f"] Oct 29 23:01:10.034813 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034797 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c789c8b7-746a-43e3-b087-0f8b291cb446" containerName="maas-api" Oct 29 23:01:10.034881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034814 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c789c8b7-746a-43e3-b087-0f8b291cb446" containerName="maas-api" Oct 29 23:01:10.034881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034839 5073 cpu_manager.go:414] "containerMap: removing container" podUID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" containerName="main" Oct 29 23:01:10.034881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034844 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" containerName="main" Oct 29 23:01:10.034981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034949 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c789c8b7-746a-43e3-b087-0f8b291cb446" containerName="maas-api" Oct 29 23:01:10.034981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.034961 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" containerName="main" Oct 29 23:01:10.038732 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.038715 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.057462 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.057436 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2ae845-openshift-default-dockercfg-5hvm9\"" Oct 29 23:01:10.074806 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.074782 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f"] Oct 29 23:01:10.133698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.133667 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.133698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.133700 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrkzn\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.133904 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.133776 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.133904 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.133894 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.134023 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.134009 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.134067 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.134032 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.134067 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.134061 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.134199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.134127 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.134300 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.134271 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.235937 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.235894 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.235952 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.235990 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236036 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236109 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236354 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236383 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkzn\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236430 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236494 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236593 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236643 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.236905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236854 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.237045 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236910 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.237045 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.236975 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.239851 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.239823 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.240777 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.240747 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.260355 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.260325 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.261504 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.261483 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkzn\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.353267 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.353210 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:10.417257 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.415648 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" event={"ID":"20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab","Type":"ContainerDied","Data":"31e5c33beac23b5eed5a5a55f0fc942dade3b33ccdc7918f81b5e9fc6b8ac1a3"} Oct 29 23:01:10.417257 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.415705 5073 scope.go:122] "RemoveContainer" containerID="eebd23514edd354826d3e704d80a093528f3e9f930c0de869ff756af8f469ae6" Oct 29 23:01:10.417257 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.415879 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2" Oct 29 23:01:10.508114 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.508087 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2"] Oct 29 23:01:10.513367 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.513340 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e803b6/auth-test-model-e803b6-kserve-5c8dd7794f-qzzd2"] Oct 29 23:01:10.694434 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:01:10.694408 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4c6d2a_b299_45a9_a9c9_06497966fe7f.slice/crio-38b7149fc85ede4b2d5dd6e6693dbb323d854033c504b9ddb5e1006aaa1fde57 WatchSource:0}: Error finding container 38b7149fc85ede4b2d5dd6e6693dbb323d854033c504b9ddb5e1006aaa1fde57: Status 404 returned error can't find the container with id 38b7149fc85ede4b2d5dd6e6693dbb323d854033c504b9ddb5e1006aaa1fde57 Oct 29 23:01:10.695042 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.695017 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f"] Oct 29 23:01:10.697082 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.697050 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:10.697185 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.697114 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:10.697185 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.697144 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:10.843405 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:10.843369 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab" path="/var/lib/kubelet/pods/20c8abc4-2229-4a9d-ab4d-f57b4a79a5ab/volumes" Oct 29 23:01:11.439177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:11.430178 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" event={"ID":"0d4c6d2a-b299-45a9-a9c9-06497966fe7f","Type":"ContainerStarted","Data":"98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a"} Oct 29 23:01:11.439177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:11.430246 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" event={"ID":"0d4c6d2a-b299-45a9-a9c9-06497966fe7f","Type":"ContainerStarted","Data":"38b7149fc85ede4b2d5dd6e6693dbb323d854033c504b9ddb5e1006aaa1fde57"} Oct 29 23:01:11.553014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:11.552962 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podStartSLOduration=2.552946072 podStartE2EDuration="2.552946072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:11.537638645 +0000 UTC m=+2786.853032727" watchObservedRunningTime="2026-10-29 23:01:11.552946072 +0000 UTC m=+2786.868340225" Oct 29 23:01:11.946266 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:11.946219 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw"] Oct 29 23:01:11.946522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:11.946472 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" podUID="351d38c6-6670-4848-be61-2b55171cf466" containerName="maas-api" containerID="cri-o://7437314622cfe477039ed4a1a5126056c2f9a608046acb9492322b6481ea2d7a" gracePeriod=30 Oct 29 23:01:12.383175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.372369 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:12.579226 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.578226 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:01:12.579226 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.578288 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:01:12.688954 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.688904 5073 generic.go:356] "Generic (PLEG): container finished" podID="351d38c6-6670-4848-be61-2b55171cf466" containerID="7437314622cfe477039ed4a1a5126056c2f9a608046acb9492322b6481ea2d7a" exitCode=0 Oct 29 23:01:12.690168 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.690114 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" event={"ID":"351d38c6-6670-4848-be61-2b55171cf466","Type":"ContainerDied","Data":"7437314622cfe477039ed4a1a5126056c2f9a608046acb9492322b6481ea2d7a"} Oct 29 23:01:12.700562 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.700538 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 23:01:12.784079 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.784046 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca\") pod \"351d38c6-6670-4848-be61-2b55171cf466\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " Oct 29 23:01:12.784261 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.784127 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\") pod \"351d38c6-6670-4848-be61-2b55171cf466\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " Oct 29 23:01:12.784456 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.784435 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s\" (UniqueName: \"kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s\") pod \"351d38c6-6670-4848-be61-2b55171cf466\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " Oct 29 23:01:12.784512 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.784477 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca" pod "351d38c6-6670-4848-be61-2b55171cf466" (UID: "351d38c6-6670-4848-be61-2b55171cf466"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:12.784551 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.784516 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs\") pod \"351d38c6-6670-4848-be61-2b55171cf466\" (UID: \"351d38c6-6670-4848-be61-2b55171cf466\") " Oct 29 23:01:12.785293 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.785268 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351d38c6-6670-4848-be61-2b55171cf466-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:12.786681 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.786652 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls" pod "351d38c6-6670-4848-be61-2b55171cf466" (UID: "351d38c6-6670-4848-be61-2b55171cf466"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:12.787138 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.787109 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s" pod "351d38c6-6670-4848-be61-2b55171cf466" (UID: "351d38c6-6670-4848-be61-2b55171cf466"). InnerVolumeSpecName "kube-api-access-nq47s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:12.787596 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.787577 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs" pod "351d38c6-6670-4848-be61-2b55171cf466" (UID: "351d38c6-6670-4848-be61-2b55171cf466"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:12.887119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.887092 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq47s\" (UniqueName: \"kubernetes.io/projected/351d38c6-6670-4848-be61-2b55171cf466-kube-api-access-nq47s\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:12.887119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.887121 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:12.887318 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:12.887131 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351d38c6-6670-4848-be61-2b55171cf466-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:13.363651 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.363614 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:01:13.363910 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.363880 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:01:13.686794 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.686718 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6"] Oct 29 23:01:13.687468 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.687450 5073 cpu_manager.go:414] "containerMap: removing container" podUID="351d38c6-6670-4848-be61-2b55171cf466" containerName="maas-api" Oct 29 23:01:13.687523 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.687472 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351d38c6-6670-4848-be61-2b55171cf466" containerName="maas-api" Oct 29 23:01:13.687612 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.687603 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="351d38c6-6670-4848-be61-2b55171cf466" containerName="maas-api" Oct 29 23:01:13.691652 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.691629 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.697844 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.697820 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" Oct 29 23:01:13.697981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.697853 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" event={"ID":"351d38c6-6670-4848-be61-2b55171cf466","Type":"ContainerDied","Data":"22c92e2fd14f80b89e1cf847f3d358ae5d5c42c510cd9f4563ed44e0ef73a10f"} Oct 29 23:01:13.697981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.697929 5073 scope.go:122] "RemoveContainer" containerID="7437314622cfe477039ed4a1a5126056c2f9a608046acb9492322b6481ea2d7a" Oct 29 23:01:13.727497 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.727466 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-36b46d-openshift-default-dockercfg-9qfvs\"" Oct 29 23:01:13.747936 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.747891 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6"] Oct 29 23:01:13.805278 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805243 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.805441 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805290 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.805441 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805387 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.805563 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805515 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.805642 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805615 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.805716 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805693 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6z6m\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.805842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805826 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.806013 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.805994 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.806105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.806052 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.839746 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.839707 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw"] Oct 29 23:01:13.855261 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.855235 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw"] Oct 29 23:01:13.907593 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907561 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.907779 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907740 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.907779 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907771 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.907919 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907824 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.907919 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907872 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.907919 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907906 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.908078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.907964 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6z6m\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.908078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.908034 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.916221 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.908183 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.916221 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.908207 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.933174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.932235 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.933174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.932974 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.933374 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.933332 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.934115 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.934091 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.934845 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.934821 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.936730 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.936694 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.937100 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.937058 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6z6m\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:13.937260 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:13.937238 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:14.032174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.031334 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:14.353908 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.353863 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:01:14.354062 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.353927 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:01:14.399747 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.399719 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6"] Oct 29 23:01:14.401019 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:01:14.400991 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7448d27_6f4d_4020_9525_7f82d08a3806.slice/crio-06bd11b8a48790e05edf6b86aa604dc9a2b0c0ca556e1b35aa654da890a67311 WatchSource:0}: Error finding container 06bd11b8a48790e05edf6b86aa604dc9a2b0c0ca556e1b35aa654da890a67311: Status 404 returned error can't find the container with id 06bd11b8a48790e05edf6b86aa604dc9a2b0c0ca556e1b35aa654da890a67311 Oct 29 23:01:14.404487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.404446 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:14.404584 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.404532 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:14.404626 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.404577 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:01:14.707015 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.706975 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" event={"ID":"a7448d27-6f4d-4020-9525-7f82d08a3806","Type":"ContainerStarted","Data":"9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba"} Oct 29 23:01:14.707535 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.707047 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" event={"ID":"a7448d27-6f4d-4020-9525-7f82d08a3806","Type":"ContainerStarted","Data":"06bd11b8a48790e05edf6b86aa604dc9a2b0c0ca556e1b35aa654da890a67311"} Oct 29 23:01:14.776509 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.776449 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podStartSLOduration=1.776432551 podStartE2EDuration="1.776432551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:14.773340355 +0000 UTC m=+2790.088734435" watchObservedRunningTime="2026-10-29 23:01:14.776432551 +0000 UTC m=+2790.091826674" Oct 29 23:01:14.846610 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:14.846574 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351d38c6-6670-4848-be61-2b55171cf466" path="/var/lib/kubelet/pods/351d38c6-6670-4848-be61-2b55171cf466/volumes" Oct 29 23:01:15.032543 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:15.032457 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:15.034578 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:15.034552 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:15.034736 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:15.034599 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:15.354204 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:15.354115 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:01:15.354373 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:15.354220 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:01:15.605015 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:15.604916 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-329f7b-7ccdff9c7f-84wzw" podUID="351d38c6-6670-4848-be61-2b55171cf466" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.154:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:01:16.068369 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:16.068328 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.069979 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:16.069865 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:16.360238 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:16.358452 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.360238 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:16.358535 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:01:17.036176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.032547 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:17.036176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.032618 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:17.630177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.608346 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:17.630177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.609817 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:17.636230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.633435 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:01:17.970198 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.967583 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4"] Oct 29 23:01:17.970198 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:17.967837 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" containerID="cri-o://b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a" gracePeriod=30 Oct 29 23:01:18.033500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:18.033465 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:18.033666 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:18.033520 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:19.033178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:19.032480 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:19.033178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:19.032540 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:20.072634 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.072592 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.073169 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.072659 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:20.697137 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.697092 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm"] Oct 29 23:01:20.705479 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.705456 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:20.738265 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.738236 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-14b0c3\"" Oct 29 23:01:20.738620 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.738601 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-14b0c3-dockercfg-w77ts\"" Oct 29 23:01:20.749842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.747562 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm"] Oct 29 23:01:20.795861 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.795822 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:20.796036 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.795887 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqkn\" (UniqueName: \"kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:20.897230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.897194 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:20.897230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.897234 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqkn\" (UniqueName: \"kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:20.898011 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.897985 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:20.940092 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:20.940060 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqkn\" (UniqueName: \"kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:21.033560 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.033518 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:01:21.033750 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.033592 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:01:21.055742 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.055714 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:21.437929 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:01:21.437892 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972766dc_cfeb_425c_ab30_64188633483b.slice/crio-8abd484ef79833ec67c988152e368668b4e61fe97428063ecf22dc6b2bfc3d2e WatchSource:0}: Error finding container 8abd484ef79833ec67c988152e368668b4e61fe97428063ecf22dc6b2bfc3d2e: Status 404 returned error can't find the container with id 8abd484ef79833ec67c988152e368668b4e61fe97428063ecf22dc6b2bfc3d2e Oct 29 23:01:21.438408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.438387 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm"] Oct 29 23:01:21.820172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.820085 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" event={"ID":"972766dc-cfeb-425c-ab30-64188633483b","Type":"ContainerStarted","Data":"9d5d0916510bb9679d16a0965eb764bd6cac5bc93d86621de5c356dd545805c4"} Oct 29 23:01:21.820172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.820175 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" event={"ID":"972766dc-cfeb-425c-ab30-64188633483b","Type":"ContainerStarted","Data":"8abd484ef79833ec67c988152e368668b4e61fe97428063ecf22dc6b2bfc3d2e"} Oct 29 23:01:21.820391 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:21.820208 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:22.038100 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:22.038066 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:22.038530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:22.038503 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:22.039186 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:22.039167 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:01:22.104256 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:22.103866 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" podStartSLOduration=2.103842252 podStartE2EDuration="2.103842252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:21.874347325 +0000 UTC m=+2797.189741450" watchObservedRunningTime="2026-10-29 23:01:22.103842252 +0000 UTC m=+2797.419236333" Oct 29 23:01:22.969001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:22.968965 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx"] Oct 29 23:01:22.969501 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:22.969222 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" containerID="cri-o://b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a" gracePeriod=30 Oct 29 23:01:23.284865 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.284826 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:23.285214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.285179 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:23.393369 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.393340 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 23:01:23.496614 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496582 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.496774 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496643 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.496774 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496765 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.496917 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496811 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.496917 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496849 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.496917 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496882 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.497085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496974 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.497085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.496985 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.497213 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497116 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.497278 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497247 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.497344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497270 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.497344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497290 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token\") pod \"b271344f-e086-4aca-a1d8-8294ea12d756\" (UID: \"b271344f-e086-4aca-a1d8-8294ea12d756\") " Oct 29 23:01:23.497525 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497491 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:23.497655 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497571 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.497981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497962 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.498085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.497985 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.498085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.498000 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.498085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.498013 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.498085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.498026 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b271344f-e086-4aca-a1d8-8294ea12d756-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.500170 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.500116 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.500287 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.500188 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "kube-api-access-x2f98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:23.500447 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.500425 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:23.502209 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.502120 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token" pod "b271344f-e086-4aca-a1d8-8294ea12d756" (UID: "b271344f-e086-4aca-a1d8-8294ea12d756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:23.599514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.599423 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b271344f-e086-4aca-a1d8-8294ea12d756-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.599514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.599465 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2f98\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-kube-api-access-x2f98\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.599514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.599478 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b271344f-e086-4aca-a1d8-8294ea12d756-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.599514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.599491 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b271344f-e086-4aca-a1d8-8294ea12d756-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.777341 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.777305 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6"] Oct 29 23:01:23.777674 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.777548 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" podUID="20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" containerName="payload-processing" containerID="cri-o://58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d" gracePeriod=30 Oct 29 23:01:23.852217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.850306 5073 generic.go:356] "Generic (PLEG): container finished" podID="b271344f-e086-4aca-a1d8-8294ea12d756" containerID="b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a" exitCode=0 Oct 29 23:01:23.852217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.851192 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" Oct 29 23:01:23.854348 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.853421 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" event={"ID":"b271344f-e086-4aca-a1d8-8294ea12d756","Type":"ContainerDied","Data":"b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a"} Oct 29 23:01:23.854348 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.853478 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4" event={"ID":"b271344f-e086-4aca-a1d8-8294ea12d756","Type":"ContainerDied","Data":"bb76927ccd2581a2b46c835bbbbb89c08e99181595eea296a2d291a2ea1f9e9f"} Oct 29 23:01:23.854348 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.853510 5073 scope.go:122] "RemoveContainer" containerID="b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a" Oct 29 23:01:23.897667 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.897644 5073 scope.go:122] "RemoveContainer" containerID="b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a" Oct 29 23:01:23.898192 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:01:23.898065 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a\": container with ID starting with b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a not found: ID does not exist" containerID="b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a" Oct 29 23:01:23.898192 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.898111 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a"} err="failed to get container status \"b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a\": rpc error: code = NotFound desc = could not find container \"b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a\": container with ID starting with b1f86ab864141ea52a79b9f86ed5e572d89f8db6695d310a2571ffd0cd7b2e1a not found: ID does not exist" Oct 29 23:01:23.934297 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.934238 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4"] Oct 29 23:01:23.939205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:23.939097 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-nl5t4"] Oct 29 23:01:24.065946 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.065923 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 23:01:24.108557 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.107693 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm\" (UniqueName: \"kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm\") pod \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " Oct 29 23:01:24.108557 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.107878 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume\") pod \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\" (UID: \"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6\") " Oct 29 23:01:24.108557 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.108453 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume" pod "20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" (UID: "20c8d02e-aa88-4237-bc6e-d18b28ffaeb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:24.108871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.108775 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.130642 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.127647 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm" pod "20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" (UID: "20c8d02e-aa88-4237-bc6e-d18b28ffaeb6"). InnerVolumeSpecName "kube-api-access-qshqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:24.242178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.222788 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshqm\" (UniqueName: \"kubernetes.io/projected/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6-kube-api-access-qshqm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.841511 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.841479 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" path="/var/lib/kubelet/pods/b271344f-e086-4aca-a1d8-8294ea12d756/volumes" Oct 29 23:01:24.859001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.858971 5073 generic.go:356] "Generic (PLEG): container finished" podID="20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" containerID="58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d" exitCode=0 Oct 29 23:01:24.859129 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.859030 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" Oct 29 23:01:24.859129 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.859054 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" event={"ID":"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6","Type":"ContainerDied","Data":"58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d"} Oct 29 23:01:24.859129 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.859113 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6" event={"ID":"20c8d02e-aa88-4237-bc6e-d18b28ffaeb6","Type":"ContainerDied","Data":"2b3fee2e989c1fe5af18757cf7330e1addcc9d3d5b664fe76358ed812b267f6b"} Oct 29 23:01:24.859273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.859136 5073 scope.go:122] "RemoveContainer" containerID="58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d" Oct 29 23:01:24.874833 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.874816 5073 scope.go:122] "RemoveContainer" containerID="58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d" Oct 29 23:01:24.875122 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:01:24.875103 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d\": container with ID starting with 58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d not found: ID does not exist" containerID="58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d" Oct 29 23:01:24.875194 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.875134 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d"} err="failed to get container status \"58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d\": rpc error: code = NotFound desc = could not find container \"58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d\": container with ID starting with 58a65a27c27dae9617a59c91549ce5d208b88a1a3bd0ed4e10897a2804a9d05d not found: ID does not exist" Oct 29 23:01:24.927893 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.927863 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6"] Oct 29 23:01:24.936263 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:24.936237 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-cf5bc884-7dvv6"] Oct 29 23:01:26.443259 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.443225 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5"] Oct 29 23:01:26.444284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.444259 5073 cpu_manager.go:414] "containerMap: removing container" podUID="20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" containerName="payload-processing" Oct 29 23:01:26.444385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.444286 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" containerName="payload-processing" Oct 29 23:01:26.444385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.444351 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" Oct 29 23:01:26.444385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.444360 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" Oct 29 23:01:26.444560 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.444544 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b271344f-e086-4aca-a1d8-8294ea12d756" containerName="istio-proxy" Oct 29 23:01:26.444615 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.444568 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" containerName="payload-processing" Oct 29 23:01:26.450184 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.450134 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.469071 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.469048 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1dabb8\"/\"kube-root-ca.crt\"" Oct 29 23:01:26.472721 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.472697 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5"] Oct 29 23:01:26.483039 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.483015 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1dabb8\"/\"openshift-service-ca.crt\"" Oct 29 23:01:26.483996 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.483966 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1dabb8\"/\"default-dockercfg-mrrhg\"" Oct 29 23:01:26.496828 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.496808 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1dabb8\"/\"ipp-route-1dabb8-kserve-self-signed-certs\"" Oct 29 23:01:26.574902 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.574864 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.575054 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.574949 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.575216 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.575193 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.575340 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.575322 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.575388 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.575371 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjc9q\" (UniqueName: \"kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.575513 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.575498 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.676891 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.676855 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677086 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.676908 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjc9q\" (UniqueName: \"kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677215 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677079 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677375 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677341 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677450 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677414 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677459 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677515 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677675 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677662 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.677996 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.677974 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.680461 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.680436 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.680672 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.680648 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.705988 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.705883 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjc9q\" (UniqueName: \"kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q\") pod \"ipp-route-1dabb8-kserve-bbcbbff7c-52mg5\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.765507 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.765463 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:26.895138 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:26.895082 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c8d02e-aa88-4237-bc6e-d18b28ffaeb6" path="/var/lib/kubelet/pods/20c8d02e-aa88-4237-bc6e-d18b28ffaeb6/volumes" Oct 29 23:01:27.236950 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:27.236920 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5"] Oct 29 23:01:27.237317 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:01:27.237294 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93100a9f_ba38_4855_99a4_527321296c7b.slice/crio-d5351ef8d21841aaef6c16e10034c447a9e9fc6ea5574bf4a919fc8d9012e9f7 WatchSource:0}: Error finding container d5351ef8d21841aaef6c16e10034c447a9e9fc6ea5574bf4a919fc8d9012e9f7: Status 404 returned error can't find the container with id d5351ef8d21841aaef6c16e10034c447a9e9fc6ea5574bf4a919fc8d9012e9f7 Oct 29 23:01:27.623537 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:27.623462 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:01:27.909552 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:27.909452 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" event={"ID":"93100a9f-ba38-4855-99a4-527321296c7b","Type":"ContainerStarted","Data":"f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44"} Oct 29 23:01:27.909552 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:27.909501 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" event={"ID":"93100a9f-ba38-4855-99a4-527321296c7b","Type":"ContainerStarted","Data":"d5351ef8d21841aaef6c16e10034c447a9e9fc6ea5574bf4a919fc8d9012e9f7"} Oct 29 23:01:27.976561 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:27.976497 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" podStartSLOduration=1.976456856 podStartE2EDuration="1.976456856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:27.956032714 +0000 UTC m=+2803.271426794" watchObservedRunningTime="2026-10-29 23:01:27.976456856 +0000 UTC m=+2803.291850937" Oct 29 23:01:28.365391 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.365363 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 23:01:28.475988 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.475945 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.476258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.476019 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.476258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.476068 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.476258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.476207 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.476692 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.476327 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.497620 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.497566 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:28.498207 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.498123 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.500933 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.500894 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.501090 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.501032 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.501375 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.501338 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.501583 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.501565 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.501938 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.501917 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy\") pod \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\" (UID: \"5ddf42fd-58e3-46a1-9f57-8bebecf625a5\") " Oct 29 23:01:28.504101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.501429 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.504101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.503563 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.504101 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504078 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "kube-api-access-99dhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.504368 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504346 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.504465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504370 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.504465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504383 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dhw\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-kube-api-access-99dhw\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.504465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504397 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.504465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504409 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.504465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.504420 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.517632 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.517595 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:28.518036 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.518012 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.518426 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.518401 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy" pod "5ddf42fd-58e3-46a1-9f57-8bebecf625a5" (UID: "5ddf42fd-58e3-46a1-9f57-8bebecf625a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.606318 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.606285 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.606318 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.606322 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.606467 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.606335 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf42fd-58e3-46a1-9f57-8bebecf625a5-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.767505 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:28.767438 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:29.276909 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.276872 5073 generic.go:356] "Generic (PLEG): container finished" podID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerID="b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a" exitCode=0 Oct 29 23:01:29.286177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.278296 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" Oct 29 23:01:29.303064 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.286482 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" event={"ID":"5ddf42fd-58e3-46a1-9f57-8bebecf625a5","Type":"ContainerDied","Data":"b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a"} Oct 29 23:01:29.303064 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.301077 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx" event={"ID":"5ddf42fd-58e3-46a1-9f57-8bebecf625a5","Type":"ContainerDied","Data":"d034f0dd52254b3881747314cb656e4b09ce19c2adf0ae07bb55f7ae9fc2592b"} Oct 29 23:01:29.303064 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.301122 5073 scope.go:122] "RemoveContainer" containerID="b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a" Oct 29 23:01:29.425940 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.425911 5073 scope.go:122] "RemoveContainer" containerID="b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a" Oct 29 23:01:29.450173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.444246 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx"] Oct 29 23:01:29.450173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.446932 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-46pqx"] Oct 29 23:01:29.458459 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:01:29.454428 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a\": container with ID starting with b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a not found: ID does not exist" containerID="b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a" Oct 29 23:01:29.458459 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:29.454470 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a"} err="failed to get container status \"b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a\": rpc error: code = NotFound desc = could not find container \"b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a\": container with ID starting with b756f43c6ede57006e9a039f089fe0f29a93b87ab419b41638e0d639d7c79b9a not found: ID does not exist" Oct 29 23:01:30.839498 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:30.839463 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" path="/var/lib/kubelet/pods/5ddf42fd-58e3-46a1-9f57-8bebecf625a5/volumes" Oct 29 23:01:32.829315 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:32.829287 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:01:36.766425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:36.766392 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:36.779912 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:36.779885 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:37.395432 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:37.395399 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:01:39.256811 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:01:39.256772 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:01:46.928749 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:46.928707 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt"] Oct 29 23:01:46.929136 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:46.928920 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" podUID="20b62beb-82c4-4577-bf93-cc54642f96f9" containerName="maas-api" containerID="cri-o://c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61" gracePeriod=30 Oct 29 23:01:47.359087 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.358788 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 23:01:47.521784 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.521742 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca\") pod \"20b62beb-82c4-4577-bf93-cc54642f96f9\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " Oct 29 23:01:47.522658 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.522479 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca" pod "20b62beb-82c4-4577-bf93-cc54642f96f9" (UID: "20b62beb-82c4-4577-bf93-cc54642f96f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:47.522935 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.522916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls\") pod \"20b62beb-82c4-4577-bf93-cc54642f96f9\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " Oct 29 23:01:47.523563 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.523546 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb\" (UniqueName: \"kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb\") pod \"20b62beb-82c4-4577-bf93-cc54642f96f9\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " Oct 29 23:01:47.566741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.560208 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs\") pod \"20b62beb-82c4-4577-bf93-cc54642f96f9\" (UID: \"20b62beb-82c4-4577-bf93-cc54642f96f9\") " Oct 29 23:01:47.566741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.561119 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b62beb-82c4-4577-bf93-cc54642f96f9-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:47.575567 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.575533 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs" pod "20b62beb-82c4-4577-bf93-cc54642f96f9" (UID: "20b62beb-82c4-4577-bf93-cc54642f96f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:47.578325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.578266 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls" pod "20b62beb-82c4-4577-bf93-cc54642f96f9" (UID: "20b62beb-82c4-4577-bf93-cc54642f96f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:47.615665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.615627 5073 generic.go:356] "Generic (PLEG): container finished" podID="20b62beb-82c4-4577-bf93-cc54642f96f9" containerID="c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61" exitCode=0 Oct 29 23:01:47.615857 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.615683 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" Oct 29 23:01:47.615857 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.615700 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" event={"ID":"20b62beb-82c4-4577-bf93-cc54642f96f9","Type":"ContainerDied","Data":"c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61"} Oct 29 23:01:47.615857 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.615780 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt" event={"ID":"20b62beb-82c4-4577-bf93-cc54642f96f9","Type":"ContainerDied","Data":"7e4bd409f443dfe60b608735914c29929a339172315cbfd26b743c58940943a8"} Oct 29 23:01:47.615857 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.615803 5073 scope.go:122] "RemoveContainer" containerID="c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61" Oct 29 23:01:47.623269 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.623234 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb" pod "20b62beb-82c4-4577-bf93-cc54642f96f9" (UID: "20b62beb-82c4-4577-bf93-cc54642f96f9"). InnerVolumeSpecName "kube-api-access-tlfjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:47.629844 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.629822 5073 scope.go:122] "RemoveContainer" containerID="c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61" Oct 29 23:01:47.630223 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:01:47.630201 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61\": container with ID starting with c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61 not found: ID does not exist" containerID="c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61" Oct 29 23:01:47.630323 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.630235 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61"} err="failed to get container status \"c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61\": rpc error: code = NotFound desc = could not find container \"c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61\": container with ID starting with c97856915cb9133e16ea0ece052d5232769d4a7539dadb7f89a7edb27b8acf61 not found: ID does not exist" Oct 29 23:01:47.662201 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.662178 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfjb\" (UniqueName: \"kubernetes.io/projected/20b62beb-82c4-4577-bf93-cc54642f96f9-kube-api-access-tlfjb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:47.662201 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.662203 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:47.662341 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.662212 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b62beb-82c4-4577-bf93-cc54642f96f9-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:01:47.983488 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.983463 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt"] Oct 29 23:01:47.996847 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:47.996820 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-afdbe8-64b89f64b7-mlfkt"] Oct 29 23:01:48.839850 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:48.839810 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b62beb-82c4-4577-bf93-cc54642f96f9" path="/var/lib/kubelet/pods/20b62beb-82c4-4577-bf93-cc54642f96f9/volumes" Oct 29 23:01:56.872659 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:56.872392 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5"] Oct 29 23:01:56.873459 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:01:56.873402 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" containerID="cri-o://68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356" gracePeriod=30 Oct 29 23:02:02.186930 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.186906 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 23:02:02.257717 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.257684 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.257897 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.257731 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.257897 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.257885 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258022 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.257928 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258022 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.257996 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258025 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.258126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258082 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258110 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258170 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258228 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket\") pod \"66145ed4-2278-4025-ac12-639607aeae8f\" (UID: \"66145ed4-2278-4025-ac12-639607aeae8f\") " Oct 29 23:02:02.258319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258250 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.258319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258301 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.258630 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.258575 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:02.259061 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.259036 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.259211 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.259193 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.259290 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.259219 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.259290 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.259234 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66145ed4-2278-4025-ac12-639607aeae8f-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.259290 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.259248 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.259290 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.259263 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.260431 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.260403 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:02.260743 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.260721 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "kube-api-access-4j7fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:02.261035 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.261015 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:02.261242 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.261214 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token" pod "66145ed4-2278-4025-ac12-639607aeae8f" (UID: "66145ed4-2278-4025-ac12-639607aeae8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:02.361278 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.361244 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66145ed4-2278-4025-ac12-639607aeae8f-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.361278 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.361273 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7fn\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-kube-api-access-4j7fn\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.361539 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.361285 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66145ed4-2278-4025-ac12-639607aeae8f-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.361539 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.361298 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66145ed4-2278-4025-ac12-639607aeae8f-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.726508 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.726251 5073 generic.go:356] "Generic (PLEG): container finished" podID="66145ed4-2278-4025-ac12-639607aeae8f" containerID="68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356" exitCode=0 Oct 29 23:02:02.726508 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.726340 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" event={"ID":"66145ed4-2278-4025-ac12-639607aeae8f","Type":"ContainerDied","Data":"68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356"} Oct 29 23:02:02.726508 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.726361 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" Oct 29 23:02:02.726508 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.726429 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5" event={"ID":"66145ed4-2278-4025-ac12-639607aeae8f","Type":"ContainerDied","Data":"38b4231fcbde78e5a19039c8bcd8a9b2a0a6e4786613abc988759f863ccf78c8"} Oct 29 23:02:02.726508 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.726454 5073 scope.go:122] "RemoveContainer" containerID="68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356" Oct 29 23:02:02.737701 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.737682 5073 scope.go:122] "RemoveContainer" containerID="68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356" Oct 29 23:02:02.738026 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:02.737998 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356\": container with ID starting with 68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356 not found: ID does not exist" containerID="68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356" Oct 29 23:02:02.738117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.738038 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356"} err="failed to get container status \"68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356\": rpc error: code = NotFound desc = could not find container \"68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356\": container with ID starting with 68cd1d0c8924491218efb5ee14f74fec4a68576277a4a7667bb0cfc12ed3f356 not found: ID does not exist" Oct 29 23:02:02.786087 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.786066 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5"] Oct 29 23:02:02.799774 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.799749 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-afdbe8-openshift-default-77c5cc4d6c-ctfj5"] Oct 29 23:02:02.840060 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:02.840035 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66145ed4-2278-4025-ac12-639607aeae8f" path="/var/lib/kubelet/pods/66145ed4-2278-4025-ac12-639607aeae8f/volumes" Oct 29 23:02:08.602434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:08.602403 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:02:09.256370 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:09.256320 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:02:14.482866 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:14.482830 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs"] Oct 29 23:02:14.483393 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:14.483097 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" containerID="cri-o://6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13" gracePeriod=30 Oct 29 23:02:14.494829 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:14.494800 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9"] Oct 29 23:02:14.495056 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:14.495020 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" containerID="cri-o://b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5" gracePeriod=30 Oct 29 23:02:19.176788 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.176744 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:19.177208 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.176809 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:19.770609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.770542 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:02:19.837895 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.837859 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:02:19.857042 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857012 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857210 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857063 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857210 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857100 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857351 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857211 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857351 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857262 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857467 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857368 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857467 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857430 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857584 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857482 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:19.857584 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857497 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857584 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857525 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857602 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857660 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857710 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857793 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857834 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.857929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857877 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.857929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857892 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:19.857929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857908 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.858258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857942 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token\") pod \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\" (UID: \"c1f8dde8-ebda-49cd-b14a-825bea2f48cd\") " Oct 29 23:02:19.858258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.857969 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy\") pod \"eca898de-4510-4f38-b7f9-bfb6a32cb466\" (UID: \"eca898de-4510-4f38-b7f9-bfb6a32cb466\") " Oct 29 23:02:19.858402 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858355 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.858466 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858420 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.858867 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858610 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.858966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858880 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.858966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858902 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca898de-4510-4f38-b7f9-bfb6a32cb466-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.858966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858916 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.858966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858929 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.858966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.858941 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.859840 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.859382 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.859949 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.859907 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.860507 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.860415 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.860607 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.860524 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.860771 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.860737 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.861640 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.861612 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.862052 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.862027 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:19.862304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.862221 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "kube-api-access-fljw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:19.862425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.862398 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:19.862674 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.862649 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:19.863647 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.863623 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "kube-api-access-dmvp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:19.863729 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.863633 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy" pod "eca898de-4510-4f38-b7f9-bfb6a32cb466" (UID: "eca898de-4510-4f38-b7f9-bfb6a32cb466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:19.864222 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.864192 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token" pod "c1f8dde8-ebda-49cd-b14a-825bea2f48cd" (UID: "c1f8dde8-ebda-49cd-b14a-825bea2f48cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:19.932188 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.932123 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz"] Oct 29 23:02:19.950331 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.950299 5073 generic.go:356] "Generic (PLEG): container finished" podID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerID="6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13" exitCode=0 Oct 29 23:02:19.950476 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.950341 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" Oct 29 23:02:19.950476 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.950379 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" event={"ID":"c1f8dde8-ebda-49cd-b14a-825bea2f48cd","Type":"ContainerDied","Data":"6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13"} Oct 29 23:02:19.950476 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.950434 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs" event={"ID":"c1f8dde8-ebda-49cd-b14a-825bea2f48cd","Type":"ContainerDied","Data":"d8e76ce59109c204a9f7d0c387b10e5ae212bac0c3d0f225d8f96978efab3c3e"} Oct 29 23:02:19.950476 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.950458 5073 scope.go:122] "RemoveContainer" containerID="6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13" Oct 29 23:02:19.952639 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.952572 5073 generic.go:356] "Generic (PLEG): container finished" podID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerID="b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5" exitCode=0 Oct 29 23:02:19.952639 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.952613 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" Oct 29 23:02:19.952639 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.952622 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" event={"ID":"eca898de-4510-4f38-b7f9-bfb6a32cb466","Type":"ContainerDied","Data":"b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5"} Oct 29 23:02:19.952639 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.952660 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9" event={"ID":"eca898de-4510-4f38-b7f9-bfb6a32cb466","Type":"ContainerDied","Data":"95734577edbaa077d7c0a1d82900bf9948076b125593f9efba5bfe0fd24c8ace"} Oct 29 23:02:19.959810 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959789 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959814 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959827 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959838 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959850 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959864 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959878 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmvp7\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-kube-api-access-dmvp7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959893 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959905 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljw9\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-kube-api-access-fljw9\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.959916 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959917 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f8dde8-ebda-49cd-b14a-825bea2f48cd-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.960761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959930 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.960761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959942 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca898de-4510-4f38-b7f9-bfb6a32cb466-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.960761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.959954 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca898de-4510-4f38-b7f9-bfb6a32cb466-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.964825 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.964741 5073 scope.go:122] "RemoveContainer" containerID="6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13" Oct 29 23:02:19.965139 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:19.965109 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13\": container with ID starting with 6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13 not found: ID does not exist" containerID="6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13" Oct 29 23:02:19.965273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.965166 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13"} err="failed to get container status \"6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13\": rpc error: code = NotFound desc = could not find container \"6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13\": container with ID starting with 6ac9995ee7f89062cb26f27f4a2c8a508eb1e0e7ed156d2807340f76c5b5de13 not found: ID does not exist" Oct 29 23:02:19.965273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.965191 5073 scope.go:122] "RemoveContainer" containerID="b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5" Oct 29 23:02:19.975319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.975294 5073 scope.go:122] "RemoveContainer" containerID="b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5" Oct 29 23:02:19.975628 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:19.975603 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5\": container with ID starting with b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5 not found: ID does not exist" containerID="b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5" Oct 29 23:02:19.975703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:19.975639 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5"} err="failed to get container status \"b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5\": rpc error: code = NotFound desc = could not find container \"b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5\": container with ID starting with b5777a031e7238daeece8b62d67c625c506cd47a4b48ede81fb47b220d47ffa5 not found: ID does not exist" Oct 29 23:02:20.022219 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:20.022123 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9"] Oct 29 23:02:20.025485 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:20.025464 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-5rts9"] Oct 29 23:02:20.066878 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:20.066837 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs"] Oct 29 23:02:20.071806 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:20.071782 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-wpqbs"] Oct 29 23:02:20.839099 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:20.839025 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" path="/var/lib/kubelet/pods/c1f8dde8-ebda-49cd-b14a-825bea2f48cd/volumes" Oct 29 23:02:20.839563 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:20.839545 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" path="/var/lib/kubelet/pods/eca898de-4510-4f38-b7f9-bfb6a32cb466/volumes" Oct 29 23:02:34.955500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:34.955431 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" containerID="cri-o://e0812e81e71c0252117cc625d6bc72ba700455833f24fb398d15ac4a08ef9a48" gracePeriod=45 Oct 29 23:02:35.104243 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.104205 5073 generic.go:356] "Generic (PLEG): container finished" podID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerID="e0812e81e71c0252117cc625d6bc72ba700455833f24fb398d15ac4a08ef9a48" exitCode=0 Oct 29 23:02:35.104416 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.104303 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" event={"ID":"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a","Type":"ContainerDied","Data":"e0812e81e71c0252117cc625d6bc72ba700455833f24fb398d15ac4a08ef9a48"} Oct 29 23:02:35.276432 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.276405 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.395957 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache\") pod \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396030 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs\") pod \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396101 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm\") pod \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396198 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir\") pod \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396249 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home\") pod \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396296 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk\" (UniqueName: \"kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk\") pod \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\" (UID: \"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a\") " Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396309 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache" pod "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" (UID: "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396539 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir" pod "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" (UID: "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.396624 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home" pod "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" (UID: "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.397104 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.397126 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.397205 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.397142 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.399222 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.399192 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm" pod "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" (UID: "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:35.399370 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.399215 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk" pod "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" (UID: "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a"). InnerVolumeSpecName "kube-api-access-cf7gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:35.400523 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.400497 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs" pod "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" (UID: "ab8ffa39-241f-4f94-9d5b-4ecdbecee27a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:35.498412 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.498328 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.498412 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.498358 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.498412 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:35.498369 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7gk\" (UniqueName: \"kubernetes.io/projected/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a-kube-api-access-cf7gk\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:36.138834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.138805 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" Oct 29 23:02:36.138834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.138819 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz" event={"ID":"ab8ffa39-241f-4f94-9d5b-4ecdbecee27a","Type":"ContainerDied","Data":"40334f135d9478014872fa49d3befa85442a8b3c6f05724a7a7ce43be7ee3b2a"} Oct 29 23:02:36.139517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.138882 5073 scope.go:122] "RemoveContainer" containerID="e0812e81e71c0252117cc625d6bc72ba700455833f24fb398d15ac4a08ef9a48" Oct 29 23:02:36.198177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.198129 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz"] Oct 29 23:02:36.232174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.232037 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-879596/premium-w1-879596-kserve-6778b6669c-jj5fz"] Oct 29 23:02:36.322712 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.321173 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb"] Oct 29 23:02:36.322712 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.321449 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" podUID="f1402654-ca43-41d9-afd6-94f7966c9783" containerName="maas-api" containerID="cri-o://827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07" gracePeriod=30 Oct 29 23:02:36.903174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.895471 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" path="/var/lib/kubelet/pods/ab8ffa39-241f-4f94-9d5b-4ecdbecee27a/volumes" Oct 29 23:02:36.903174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:36.897912 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 23:02:37.051591 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.051510 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs\") pod \"f1402654-ca43-41d9-afd6-94f7966c9783\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " Oct 29 23:02:37.051741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.051713 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca\") pod \"f1402654-ca43-41d9-afd6-94f7966c9783\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " Oct 29 23:02:37.051781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.051752 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg\") pod \"f1402654-ca43-41d9-afd6-94f7966c9783\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " Oct 29 23:02:37.052111 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.051832 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\") pod \"f1402654-ca43-41d9-afd6-94f7966c9783\" (UID: \"f1402654-ca43-41d9-afd6-94f7966c9783\") " Oct 29 23:02:37.052111 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.052042 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca" pod "f1402654-ca43-41d9-afd6-94f7966c9783" (UID: "f1402654-ca43-41d9-afd6-94f7966c9783"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:37.053442 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.053419 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1402654-ca43-41d9-afd6-94f7966c9783-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.054116 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.054093 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs" pod "f1402654-ca43-41d9-afd6-94f7966c9783" (UID: "f1402654-ca43-41d9-afd6-94f7966c9783"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:37.054713 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.054686 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg" pod "f1402654-ca43-41d9-afd6-94f7966c9783" (UID: "f1402654-ca43-41d9-afd6-94f7966c9783"). InnerVolumeSpecName "kube-api-access-rbxmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.054917 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.054895 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls" pod "f1402654-ca43-41d9-afd6-94f7966c9783" (UID: "f1402654-ca43-41d9-afd6-94f7966c9783"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:37.161171 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.160165 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/f1402654-ca43-41d9-afd6-94f7966c9783-kube-api-access-rbxmg\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.161171 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.160203 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.161171 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.160218 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1402654-ca43-41d9-afd6-94f7966c9783-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.183500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.179769 5073 generic.go:356] "Generic (PLEG): container finished" podID="f1402654-ca43-41d9-afd6-94f7966c9783" containerID="827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07" exitCode=0 Oct 29 23:02:37.183500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.179897 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" Oct 29 23:02:37.183500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.179901 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" event={"ID":"f1402654-ca43-41d9-afd6-94f7966c9783","Type":"ContainerDied","Data":"827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07"} Oct 29 23:02:37.183500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.180076 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" event={"ID":"f1402654-ca43-41d9-afd6-94f7966c9783","Type":"ContainerDied","Data":"3f646c80e25e0aa7e664578c81e50e27f3162863e0369beab06d59b526cccc85"} Oct 29 23:02:37.183500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.180112 5073 scope.go:122] "RemoveContainer" containerID="827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07" Oct 29 23:02:37.192018 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.192002 5073 scope.go:122] "RemoveContainer" containerID="827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07" Oct 29 23:02:37.192334 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:37.192312 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07\": container with ID starting with 827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07 not found: ID does not exist" containerID="827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07" Oct 29 23:02:37.192412 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.192344 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07"} err="failed to get container status \"827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07\": rpc error: code = NotFound desc = could not find container \"827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07\": container with ID starting with 827ef2803a3dbaa125f73e81398254a62d0ae6195baf4477f303403888e5fd07 not found: ID does not exist" Oct 29 23:02:37.243543 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.243514 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb"] Oct 29 23:02:37.248425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:37.248397 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb"] Oct 29 23:02:38.846709 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:38.846675 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1402654-ca43-41d9-afd6-94f7966c9783" path="/var/lib/kubelet/pods/f1402654-ca43-41d9-afd6-94f7966c9783/volumes" Oct 29 23:02:39.283842 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:39.283232 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:02:39.632236 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:39.629241 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-84304f-5c4f95f987-8nglb" podUID="f1402654-ca43-41d9-afd6-94f7966c9783" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.103:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:43.517845 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:43.516908 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": read tcp 10.132.0.2:34110->10.132.0.184:15021: read: connection reset by peer" start-of-body= Oct 29 23:02:43.517845 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:43.516971 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": read tcp 10.132.0.2:34110->10.132.0.184:15021: read: connection reset by peer" Oct 29 23:02:44.234925 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.234893 5073 generic.go:356] "Generic (PLEG): container finished" podID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerID="aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea" exitCode=0 Oct 29 23:02:44.235104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.234973 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" event={"ID":"e64aaf8d-64fe-4420-bdb5-a604c82f4305","Type":"ContainerDied","Data":"aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea"} Oct 29 23:02:44.235388 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.235369 5073 scope.go:122] "RemoveContainer" containerID="aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea" Oct 29 23:02:44.236908 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.236878 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:02:44.236976 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.236962 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:02:44.237014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.236998 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:02:44.565751 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.565667 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm"] Oct 29 23:02:44.566288 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:44.565940 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" containerID="cri-o://5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13" gracePeriod=30 Oct 29 23:02:45.246453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:45.246414 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" event={"ID":"e64aaf8d-64fe-4420-bdb5-a604c82f4305","Type":"ContainerStarted","Data":"69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78"} Oct 29 23:02:45.599287 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:45.599197 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:02:45.599675 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:45.599406 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:02:45.599675 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:45.599459 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:02:46.599767 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:46.599729 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:02:46.600203 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:46.599788 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:02:47.599454 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:47.599418 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:02:47.599629 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:47.599473 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:02:47.966482 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:47.966444 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:47.966962 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:47.966519 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:48.599932 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:48.599898 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:02:48.600112 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:48.599954 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:02:49.600142 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:49.600109 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:02:49.600517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:49.600214 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:02:49.951588 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:49.951561 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 23:02:50.069013 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.068979 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w"] Oct 29 23:02:50.075889 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.075860 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076033 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.075903 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076033 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.075958 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076033 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.075999 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076261 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.076060 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076322 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.076264 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.076318 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.076351 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076491 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.076435 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data\") pod \"8862115c-5cd4-4636-bba6-16d04553feb9\" (UID: \"8862115c-5cd4-4636-bba6-16d04553feb9\") " Oct 29 23:02:50.076970 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.076945 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.077381 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.077359 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.077855 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.077781 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.078451 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.078422 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:50.078555 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.078491 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.078555 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.078513 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.078555 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.078531 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.078855 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.078714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.079572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.079548 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:50.079814 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.079791 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "kube-api-access-69w6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:50.079886 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.079828 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:50.080667 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.080646 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy" pod "8862115c-5cd4-4636-bba6-16d04553feb9" (UID: "8862115c-5cd4-4636-bba6-16d04553feb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.180180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.180090 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.180180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.180119 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8862115c-5cd4-4636-bba6-16d04553feb9-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.180180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.180128 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.180180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.180136 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8862115c-5cd4-4636-bba6-16d04553feb9-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.180180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.180144 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69w6q\" (UniqueName: \"kubernetes.io/projected/8862115c-5cd4-4636-bba6-16d04553feb9-kube-api-access-69w6q\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.180180 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.180168 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8862115c-5cd4-4636-bba6-16d04553feb9-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.283905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.283869 5073 generic.go:356] "Generic (PLEG): container finished" podID="8862115c-5cd4-4636-bba6-16d04553feb9" containerID="5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13" exitCode=0 Oct 29 23:02:50.284105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.283932 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" Oct 29 23:02:50.284105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.283953 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" event={"ID":"8862115c-5cd4-4636-bba6-16d04553feb9","Type":"ContainerDied","Data":"5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13"} Oct 29 23:02:50.284105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.284011 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm" event={"ID":"8862115c-5cd4-4636-bba6-16d04553feb9","Type":"ContainerDied","Data":"8c6903f45e386d792a46896def128043fa4a3cefa03b9d62d5f04edb1f7e576a"} Oct 29 23:02:50.284105 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.284035 5073 scope.go:122] "RemoveContainer" containerID="5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13" Oct 29 23:02:50.301063 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.301040 5073 scope.go:122] "RemoveContainer" containerID="5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13" Oct 29 23:02:50.301377 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:02:50.301358 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13\": container with ID starting with 5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13 not found: ID does not exist" containerID="5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13" Oct 29 23:02:50.301454 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.301386 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13"} err="failed to get container status \"5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13\": rpc error: code = NotFound desc = could not find container \"5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13\": container with ID starting with 5ee1085a17e9038a0a97e779c0171e72d48566be419b65db4a4e73dc626bff13 not found: ID does not exist" Oct 29 23:02:50.402246 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.402215 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm"] Oct 29 23:02:50.405889 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.405868 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-492rm"] Oct 29 23:02:50.599382 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.599342 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:02:50.599566 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.599396 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:02:50.841268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:50.841233 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" path="/var/lib/kubelet/pods/8862115c-5cd4-4636-bba6-16d04553feb9/volumes" Oct 29 23:02:51.604558 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:51.604523 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:02:51.604941 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:51.604919 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:02:51.605577 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:51.605558 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:02:52.877700 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:52.877668 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:02:59.890365 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:02:59.890327 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz_ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e/payload-processing/0.log" Oct 29 23:03:05.308533 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.308474 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" containerID="cri-o://877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19" gracePeriod=45 Oct 29 23:03:05.660327 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.660286 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 23:03:05.751709 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.751670 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm\") pod \"053148a5-5ce2-4131-9aab-c09455de26b4\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " Oct 29 23:03:05.751975 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.751831 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home\") pod \"053148a5-5ce2-4131-9aab-c09455de26b4\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " Oct 29 23:03:05.751975 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.751874 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs\") pod \"053148a5-5ce2-4131-9aab-c09455de26b4\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " Oct 29 23:03:05.751975 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.751916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw\" (UniqueName: \"kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw\") pod \"053148a5-5ce2-4131-9aab-c09455de26b4\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " Oct 29 23:03:05.752517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.752066 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir\") pod \"053148a5-5ce2-4131-9aab-c09455de26b4\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " Oct 29 23:03:05.752517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.752124 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache\") pod \"053148a5-5ce2-4131-9aab-c09455de26b4\" (UID: \"053148a5-5ce2-4131-9aab-c09455de26b4\") " Oct 29 23:03:05.752984 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.752955 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache" pod "053148a5-5ce2-4131-9aab-c09455de26b4" (UID: "053148a5-5ce2-4131-9aab-c09455de26b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.753314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.753285 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir" pod "053148a5-5ce2-4131-9aab-c09455de26b4" (UID: "053148a5-5ce2-4131-9aab-c09455de26b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.753513 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.753496 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home" pod "053148a5-5ce2-4131-9aab-c09455de26b4" (UID: "053148a5-5ce2-4131-9aab-c09455de26b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.756126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.756098 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw" pod "053148a5-5ce2-4131-9aab-c09455de26b4" (UID: "053148a5-5ce2-4131-9aab-c09455de26b4"). InnerVolumeSpecName "kube-api-access-k4pmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.756860 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.756836 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs" pod "053148a5-5ce2-4131-9aab-c09455de26b4" (UID: "053148a5-5ce2-4131-9aab-c09455de26b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.757217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.757194 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm" pod "053148a5-5ce2-4131-9aab-c09455de26b4" (UID: "053148a5-5ce2-4131-9aab-c09455de26b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.854443 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.854400 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.854443 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.854436 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053148a5-5ce2-4131-9aab-c09455de26b4-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.854443 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.854452 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4pmw\" (UniqueName: \"kubernetes.io/projected/053148a5-5ce2-4131-9aab-c09455de26b4-kube-api-access-k4pmw\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.854694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.854464 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.854694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.854477 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.854694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:05.854487 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053148a5-5ce2-4131-9aab-c09455de26b4-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:06.416797 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.416758 5073 generic.go:356] "Generic (PLEG): container finished" podID="053148a5-5ce2-4131-9aab-c09455de26b4" containerID="877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19" exitCode=0 Oct 29 23:03:06.417281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.416844 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" Oct 29 23:03:06.417281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.416845 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" event={"ID":"053148a5-5ce2-4131-9aab-c09455de26b4","Type":"ContainerDied","Data":"877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19"} Oct 29 23:03:06.417281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.416962 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w" event={"ID":"053148a5-5ce2-4131-9aab-c09455de26b4","Type":"ContainerDied","Data":"e5519f851c7f4899c81a6c716b6ae03bf82afb22e4c41c9defb0f65b8279524f"} Oct 29 23:03:06.417281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.416980 5073 scope.go:122] "RemoveContainer" containerID="877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19" Oct 29 23:03:06.429058 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.429040 5073 scope.go:122] "RemoveContainer" containerID="877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19" Oct 29 23:03:06.429354 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:06.429332 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19\": container with ID starting with 877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19 not found: ID does not exist" containerID="877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19" Oct 29 23:03:06.429406 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.429363 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19"} err="failed to get container status \"877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19\": rpc error: code = NotFound desc = could not find container \"877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19\": container with ID starting with 877cd1090e81ae2f15e6bd7b9da38c3a87fbcae56e6083e7d873b25887906a19 not found: ID does not exist" Oct 29 23:03:06.471992 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.471967 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w"] Oct 29 23:03:06.484745 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.484718 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/premium-w2-8d3632-kserve-8997cb4-7cl8w"] Oct 29 23:03:06.839683 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:06.839649 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" path="/var/lib/kubelet/pods/053148a5-5ce2-4131-9aab-c09455de26b4/volumes" Oct 29 23:03:09.257250 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:09.257210 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:03:17.003289 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:17.003245 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7"] Oct 29 23:03:17.003831 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:17.003498 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" containerID="cri-o://f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109" gracePeriod=30 Oct 29 23:03:17.891424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:17.891387 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh"] Oct 29 23:03:17.891680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:17.891647 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" podUID="2cc50559-210a-4f52-af82-4f2b78a28bcb" containerName="maas-api" containerID="cri-o://11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d" gracePeriod=30 Oct 29 23:03:17.981896 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:17.981854 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" podUID="2cc50559-210a-4f52-af82-4f2b78a28bcb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.171:8443/health\": dial tcp 10.132.0.171:8443: connect: connection refused" Oct 29 23:03:18.090830 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.090792 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm"] Oct 29 23:03:18.091659 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.091624 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" podUID="972766dc-cfeb-425c-ab30-64188633483b" containerName="payload-pre-processing" containerID="cri-o://9d5d0916510bb9679d16a0965eb764bd6cac5bc93d86621de5c356dd545805c4" gracePeriod=30 Oct 29 23:03:18.297584 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.297551 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 23:03:18.524123 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.522904 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs\") pod \"2cc50559-210a-4f52-af82-4f2b78a28bcb\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " Oct 29 23:03:18.524123 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.522962 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\") pod \"2cc50559-210a-4f52-af82-4f2b78a28bcb\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " Oct 29 23:03:18.524123 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.523047 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca\") pod \"2cc50559-210a-4f52-af82-4f2b78a28bcb\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " Oct 29 23:03:18.524123 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.523418 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca" pod "2cc50559-210a-4f52-af82-4f2b78a28bcb" (UID: "2cc50559-210a-4f52-af82-4f2b78a28bcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:18.524123 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.523545 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g\" (UniqueName: \"kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g\") pod \"2cc50559-210a-4f52-af82-4f2b78a28bcb\" (UID: \"2cc50559-210a-4f52-af82-4f2b78a28bcb\") " Oct 29 23:03:18.524531 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.524408 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc50559-210a-4f52-af82-4f2b78a28bcb-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.570441 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.559552 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls" pod "2cc50559-210a-4f52-af82-4f2b78a28bcb" (UID: "2cc50559-210a-4f52-af82-4f2b78a28bcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:18.570441 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.559705 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g" pod "2cc50559-210a-4f52-af82-4f2b78a28bcb" (UID: "2cc50559-210a-4f52-af82-4f2b78a28bcb"). InnerVolumeSpecName "kube-api-access-nk85g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:18.570441 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.559755 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs" pod "2cc50559-210a-4f52-af82-4f2b78a28bcb" (UID: "2cc50559-210a-4f52-af82-4f2b78a28bcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:18.588413 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.588368 5073 generic.go:356] "Generic (PLEG): container finished" podID="972766dc-cfeb-425c-ab30-64188633483b" containerID="9d5d0916510bb9679d16a0965eb764bd6cac5bc93d86621de5c356dd545805c4" exitCode=0 Oct 29 23:03:18.588607 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.588517 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" event={"ID":"972766dc-cfeb-425c-ab30-64188633483b","Type":"ContainerDied","Data":"9d5d0916510bb9679d16a0965eb764bd6cac5bc93d86621de5c356dd545805c4"} Oct 29 23:03:18.591242 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.591206 5073 generic.go:356] "Generic (PLEG): container finished" podID="2cc50559-210a-4f52-af82-4f2b78a28bcb" containerID="11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d" exitCode=0 Oct 29 23:03:18.591397 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.591285 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" event={"ID":"2cc50559-210a-4f52-af82-4f2b78a28bcb","Type":"ContainerDied","Data":"11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d"} Oct 29 23:03:18.591397 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.591335 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" event={"ID":"2cc50559-210a-4f52-af82-4f2b78a28bcb","Type":"ContainerDied","Data":"260d0ca1d7c42fd2c4cef26f0853b4787cc6a6825cfa025564e9978f5aaa6bd9"} Oct 29 23:03:18.591397 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.591363 5073 scope.go:122] "RemoveContainer" containerID="11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d" Oct 29 23:03:18.591579 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.591543 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh" Oct 29 23:03:18.612376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.612308 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:03:18.620222 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.620196 5073 scope.go:122] "RemoveContainer" containerID="11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d" Oct 29 23:03:18.620574 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:18.620540 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d\": container with ID starting with 11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d not found: ID does not exist" containerID="11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d" Oct 29 23:03:18.620660 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.620579 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d"} err="failed to get container status \"11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d\": rpc error: code = NotFound desc = could not find container \"11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d\": container with ID starting with 11586f9ac02ae6ea8a09803519405d051dc1adb0a6043663c619b10bad7e895d not found: ID does not exist" Oct 29 23:03:18.625314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.625294 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.625314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.625315 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc50559-210a-4f52-af82-4f2b78a28bcb-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.625435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.625324 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk85g\" (UniqueName: \"kubernetes.io/projected/2cc50559-210a-4f52-af82-4f2b78a28bcb-kube-api-access-nk85g\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.659505 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.659472 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh"] Oct 29 23:03:18.663852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.663828 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-14b0c3-59dc78c8c6-hjlxh"] Oct 29 23:03:18.726323 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.726283 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn\" (UniqueName: \"kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn\") pod \"972766dc-cfeb-425c-ab30-64188633483b\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " Oct 29 23:03:18.726547 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.726525 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume\") pod \"972766dc-cfeb-425c-ab30-64188633483b\" (UID: \"972766dc-cfeb-425c-ab30-64188633483b\") " Oct 29 23:03:18.726938 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.726907 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume" pod "972766dc-cfeb-425c-ab30-64188633483b" (UID: "972766dc-cfeb-425c-ab30-64188633483b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:18.727374 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.727352 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972766dc-cfeb-425c-ab30-64188633483b-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.728955 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.728925 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn" pod "972766dc-cfeb-425c-ab30-64188633483b" (UID: "972766dc-cfeb-425c-ab30-64188633483b"). InnerVolumeSpecName "kube-api-access-rpqkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:18.829912 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.829814 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqkn\" (UniqueName: \"kubernetes.io/projected/972766dc-cfeb-425c-ab30-64188633483b-kube-api-access-rpqkn\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.843762 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:18.843726 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc50559-210a-4f52-af82-4f2b78a28bcb" path="/var/lib/kubelet/pods/2cc50559-210a-4f52-af82-4f2b78a28bcb/volumes" Oct 29 23:03:19.600306 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:19.600265 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" event={"ID":"972766dc-cfeb-425c-ab30-64188633483b","Type":"ContainerDied","Data":"8abd484ef79833ec67c988152e368668b4e61fe97428063ecf22dc6b2bfc3d2e"} Oct 29 23:03:19.600306 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:19.600306 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm" Oct 29 23:03:19.600898 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:19.600339 5073 scope.go:122] "RemoveContainer" containerID="9d5d0916510bb9679d16a0965eb764bd6cac5bc93d86621de5c356dd545805c4" Oct 29 23:03:19.654865 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:19.654839 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm"] Oct 29 23:03:19.667650 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:19.667627 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-78cd988795-b96qm"] Oct 29 23:03:20.839010 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:20.838978 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972766dc-cfeb-425c-ab30-64188633483b" path="/var/lib/kubelet/pods/972766dc-cfeb-425c-ab30-64188633483b/volumes" Oct 29 23:03:22.305385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.305350 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 23:03:22.376096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376056 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.376914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376237 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.376914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376289 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.376914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376323 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.376914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376363 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.376914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376752 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.376914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376878 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.378314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.376960 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.378314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.377133 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs\") pod \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\" (UID: \"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c\") " Oct 29 23:03:22.378314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.377656 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.378314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.377690 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.378314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.378101 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.378314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.378288 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.379804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.378341 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:22.379804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.379282 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.379804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.379306 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.379804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.379319 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.379804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.379331 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.379804 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.379361 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.381673 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.381646 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "kube-api-access-v7qn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:22.381755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.381683 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:22.381998 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.381974 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.384842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.384814 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo" pod "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" (UID: "b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:22.481048 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.481012 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.481048 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.481047 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.481340 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.481061 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7qn7\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-kube-api-access-v7qn7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.481340 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.481074 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.675850 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.673894 5073 generic.go:356] "Generic (PLEG): container finished" podID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerID="f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109" exitCode=0 Oct 29 23:03:22.675850 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.674016 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" event={"ID":"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c","Type":"ContainerDied","Data":"f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109"} Oct 29 23:03:22.675850 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.674107 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" event={"ID":"b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c","Type":"ContainerDied","Data":"2e65e2e7ef72ef4141b9c55351ded803b74028182ae3c5c0e62823154f4b83b1"} Oct 29 23:03:22.675850 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.674134 5073 scope.go:122] "RemoveContainer" containerID="f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109" Oct 29 23:03:22.675850 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.674282 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7" Oct 29 23:03:22.704830 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.704801 5073 scope.go:122] "RemoveContainer" containerID="f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109" Oct 29 23:03:22.705288 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:22.705256 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109\": container with ID starting with f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109 not found: ID does not exist" containerID="f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109" Oct 29 23:03:22.705401 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.705301 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109"} err="failed to get container status \"f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109\": rpc error: code = NotFound desc = could not find container \"f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109\": container with ID starting with f1e1526cb07cafcd777568a59cda887e958c600ea0cd3b1d448ee6d4a9ee4109 not found: ID does not exist" Oct 29 23:03:22.736025 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.735987 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7"] Oct 29 23:03:22.741542 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.741514 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6kjv7"] Oct 29 23:03:22.839753 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:22.839722 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" path="/var/lib/kubelet/pods/b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c/volumes" Oct 29 23:03:29.682014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:29.681903 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:03:39.257177 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:39.257116 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:03:41.882919 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:41.882874 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg"] Oct 29 23:03:41.883536 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:41.883211 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" containerID="cri-o://07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e" gracePeriod=30 Oct 29 23:03:43.973229 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:43.973189 5073 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:03:43.973645 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:43.973249 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:03:47.185294 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.185263 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.236679 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.236764 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.236857 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.236908 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.236946 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.236995 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.237126 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.237189 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.237665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.237241 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data\") pod \"fec36d33-ee32-4d02-b608-3170bff96d4b\" (UID: \"fec36d33-ee32-4d02-b608-3170bff96d4b\") " Oct 29 23:03:47.239203 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.238997 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.239737 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.239425 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.242117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.241827 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.242117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.242084 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:47.242648 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.242624 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.252484 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.252441 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:47.252484 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.252448 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.252615 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.252491 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:47.253974 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.253942 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt" pod "fec36d33-ee32-4d02-b608-3170bff96d4b" (UID: "fec36d33-ee32-4d02-b608-3170bff96d4b"). InnerVolumeSpecName "kube-api-access-s7bxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:47.339885 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339847 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.339885 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339880 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-kube-api-access-s7bxt\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339898 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339909 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec36d33-ee32-4d02-b608-3170bff96d4b-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339918 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339925 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339933 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339941 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec36d33-ee32-4d02-b608-3170bff96d4b-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.340096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.339948 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec36d33-ee32-4d02-b608-3170bff96d4b-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.876226 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.876190 5073 generic.go:356] "Generic (PLEG): container finished" podID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerID="07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e" exitCode=0 Oct 29 23:03:47.876425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.876255 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" event={"ID":"fec36d33-ee32-4d02-b608-3170bff96d4b","Type":"ContainerDied","Data":"07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e"} Oct 29 23:03:47.876425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.876292 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" Oct 29 23:03:47.876425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.876309 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg" event={"ID":"fec36d33-ee32-4d02-b608-3170bff96d4b","Type":"ContainerDied","Data":"99509d3958bdf3ac554f201add3be9cfda98033bd5ee205d7fc353725b23bc1e"} Oct 29 23:03:47.876425 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.876335 5073 scope.go:122] "RemoveContainer" containerID="07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e" Oct 29 23:03:47.887727 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.887703 5073 scope.go:122] "RemoveContainer" containerID="07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e" Oct 29 23:03:47.888022 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:47.887994 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e\": container with ID starting with 07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e not found: ID does not exist" containerID="07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e" Oct 29 23:03:47.888135 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.888032 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e"} err="failed to get container status \"07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e\": rpc error: code = NotFound desc = could not find container \"07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e\": container with ID starting with 07a880b0bda846c60cc5ec95dcb95fe9d58687cddf0b53c6d82bb3985a1c579e not found: ID does not exist" Oct 29 23:03:47.931497 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.931467 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg"] Oct 29 23:03:47.947706 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:47.947673 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-5rpfg"] Oct 29 23:03:48.839366 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:48.839333 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" path="/var/lib/kubelet/pods/fec36d33-ee32-4d02-b608-3170bff96d4b/volumes" Oct 29 23:03:52.963811 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:52.963775 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz"] Oct 29 23:03:52.964256 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:52.963979 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" podUID="ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" containerName="payload-processing" containerID="cri-o://d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f" gracePeriod=30 Oct 29 23:03:53.357199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.352285 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 23:03:53.436376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.436331 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d\" (UniqueName: \"kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d\") pod \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " Oct 29 23:03:53.436597 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.436421 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume\") pod \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\" (UID: \"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e\") " Oct 29 23:03:53.437112 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.437085 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume" pod "ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" (UID: "ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:53.439140 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.439117 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d" pod "ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" (UID: "ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e"). InnerVolumeSpecName "kube-api-access-gns5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:53.544549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.544427 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gns5d\" (UniqueName: \"kubernetes.io/projected/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-kube-api-access-gns5d\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.544549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.544464 5073 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e-plugins-config-volume\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.920632 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.920534 5073 generic.go:356] "Generic (PLEG): container finished" podID="ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" containerID="d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f" exitCode=0 Oct 29 23:03:53.920632 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.920584 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" Oct 29 23:03:53.920632 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.920593 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" event={"ID":"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e","Type":"ContainerDied","Data":"d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f"} Oct 29 23:03:53.920632 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.920635 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz" event={"ID":"ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e","Type":"ContainerDied","Data":"b4ef498f3b19830c3799ad3f601057630a29aac9d506658a3b60dc931a04203d"} Oct 29 23:03:53.920968 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.920656 5073 scope.go:122] "RemoveContainer" containerID="d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f" Oct 29 23:03:53.932034 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.932011 5073 scope.go:122] "RemoveContainer" containerID="d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f" Oct 29 23:03:53.932392 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:03:53.932365 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f\": container with ID starting with d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f not found: ID does not exist" containerID="d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f" Oct 29 23:03:53.932476 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:53.932405 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f"} err="failed to get container status \"d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f\": rpc error: code = NotFound desc = could not find container \"d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f\": container with ID starting with d291fed3a8b5a76d9ef338795449a4291e9dad62cf4abc25399baffc68654d6f not found: ID does not exist" Oct 29 23:03:54.009741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:54.009710 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz"] Oct 29 23:03:54.036837 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:54.036801 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1dabb8-6897c5bb8b-fwpxz"] Oct 29 23:03:54.839247 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:54.839211 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" path="/var/lib/kubelet/pods/ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e/volumes" Oct 29 23:03:56.792393 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:56.792356 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:03:59.158250 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:03:59.158211 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5"] Oct 29 23:04:09.255771 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:04:09.255725 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:04:14.185319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.185278 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" podUID="93100a9f-ba38-4855-99a4-527321296c7b" containerName="main" containerID="cri-o://f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44" gracePeriod=45 Oct 29 23:04:14.498303 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.498271 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:04:14.648842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.648805 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm\") pod \"93100a9f-ba38-4855-99a4-527321296c7b\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " Oct 29 23:04:14.649075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.648851 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir\") pod \"93100a9f-ba38-4855-99a4-527321296c7b\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " Oct 29 23:04:14.649075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.648906 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q\" (UniqueName: \"kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q\") pod \"93100a9f-ba38-4855-99a4-527321296c7b\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " Oct 29 23:04:14.649075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.648988 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache\") pod \"93100a9f-ba38-4855-99a4-527321296c7b\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " Oct 29 23:04:14.649291 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.649083 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home\") pod \"93100a9f-ba38-4855-99a4-527321296c7b\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " Oct 29 23:04:14.649291 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.649168 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs\") pod \"93100a9f-ba38-4855-99a4-527321296c7b\" (UID: \"93100a9f-ba38-4855-99a4-527321296c7b\") " Oct 29 23:04:14.652583 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.652539 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache" pod "93100a9f-ba38-4855-99a4-527321296c7b" (UID: "93100a9f-ba38-4855-99a4-527321296c7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:14.666376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.652999 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home" pod "93100a9f-ba38-4855-99a4-527321296c7b" (UID: "93100a9f-ba38-4855-99a4-527321296c7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:14.666376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.653271 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir" pod "93100a9f-ba38-4855-99a4-527321296c7b" (UID: "93100a9f-ba38-4855-99a4-527321296c7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:14.666376 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.658485 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm" pod "93100a9f-ba38-4855-99a4-527321296c7b" (UID: "93100a9f-ba38-4855-99a4-527321296c7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:14.687474 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.687409 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs" pod "93100a9f-ba38-4855-99a4-527321296c7b" (UID: "93100a9f-ba38-4855-99a4-527321296c7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:14.688323 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.688267 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q" pod "93100a9f-ba38-4855-99a4-527321296c7b" (UID: "93100a9f-ba38-4855-99a4-527321296c7b"). InnerVolumeSpecName "kube-api-access-rjc9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:14.751172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.751051 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:14.751172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.751084 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:14.751172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.751093 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93100a9f-ba38-4855-99a4-527321296c7b-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:14.751172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.751100 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:14.751172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.751131 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93100a9f-ba38-4855-99a4-527321296c7b-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:14.751172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:14.751141 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjc9q\" (UniqueName: \"kubernetes.io/projected/93100a9f-ba38-4855-99a4-527321296c7b-kube-api-access-rjc9q\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:15.083422 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.083313 5073 generic.go:356] "Generic (PLEG): container finished" podID="93100a9f-ba38-4855-99a4-527321296c7b" containerID="f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44" exitCode=0 Oct 29 23:04:15.083422 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.083377 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" Oct 29 23:04:15.083680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.083405 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" event={"ID":"93100a9f-ba38-4855-99a4-527321296c7b","Type":"ContainerDied","Data":"f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44"} Oct 29 23:04:15.083680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.083548 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" event={"ID":"93100a9f-ba38-4855-99a4-527321296c7b","Type":"ContainerDied","Data":"d5351ef8d21841aaef6c16e10034c447a9e9fc6ea5574bf4a919fc8d9012e9f7"} Oct 29 23:04:15.083680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.083573 5073 scope.go:122] "RemoveContainer" containerID="f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44" Oct 29 23:04:15.094965 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.094945 5073 scope.go:122] "RemoveContainer" containerID="f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44" Oct 29 23:04:15.095282 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:04:15.095253 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44\": container with ID starting with f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44 not found: ID does not exist" containerID="f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44" Oct 29 23:04:15.095375 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.095284 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44"} err="failed to get container status \"f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44\": rpc error: code = NotFound desc = could not find container \"f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44\": container with ID starting with f895d32d65eb0591a00cf540a9f66dc74197d797b67cf9d558686671ff2bfc44 not found: ID does not exist" Oct 29 23:04:15.138440 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.138413 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5"] Oct 29 23:04:15.151663 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.151639 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5"] Oct 29 23:04:15.376806 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:15.376702 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-1dabb8/ipp-route-1dabb8-kserve-bbcbbff7c-52mg5" podUID="93100a9f-ba38-4855-99a4-527321296c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:16.838538 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:16.838499 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93100a9f-ba38-4855-99a4-527321296c7b" path="/var/lib/kubelet/pods/93100a9f-ba38-4855-99a4-527321296c7b/volumes" Oct 29 23:04:27.660860 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:27.660818 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx"] Oct 29 23:04:27.662248 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:27.661101 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" containerID="cri-o://a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605" gracePeriod=30 Oct 29 23:04:32.726474 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:32.726444 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:32.726809 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:32.726502 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:32.960611 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:32.960586 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 23:04:33.081062 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081029 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081349 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081219 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081344 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081478 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081420 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081478 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081466 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081598 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081522 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.081598 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081533 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081598 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081594 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081764 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081626 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081764 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081694 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket\") pod \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\" (UID: \"256cffb1-0fc4-42f0-8496-4cd71c3a84c9\") " Oct 29 23:04:33.081764 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081735 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.081903 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081770 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:33.082010 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.081983 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.082173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.082137 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.082683 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.082658 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.082781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.082688 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.082781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.082707 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.082781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.082720 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.082781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.082733 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.084446 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.084418 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "kube-api-access-lnfdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.084573 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.084557 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.084679 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.084659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.084776 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.084752 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo" pod "256cffb1-0fc4-42f0-8496-4cd71c3a84c9" (UID: "256cffb1-0fc4-42f0-8496-4cd71c3a84c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:33.184426 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.184386 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfdq\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-kube-api-access-lnfdq\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.184426 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.184422 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.184426 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.184432 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.184708 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.184456 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256cffb1-0fc4-42f0-8496-4cd71c3a84c9-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.209334 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.209296 5073 generic.go:356] "Generic (PLEG): container finished" podID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerID="a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605" exitCode=0 Oct 29 23:04:33.209512 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.209361 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" Oct 29 23:04:33.209512 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.209379 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" event={"ID":"256cffb1-0fc4-42f0-8496-4cd71c3a84c9","Type":"ContainerDied","Data":"a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605"} Oct 29 23:04:33.209512 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.209472 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx" event={"ID":"256cffb1-0fc4-42f0-8496-4cd71c3a84c9","Type":"ContainerDied","Data":"bbf05a9db7935895b1e1a116d017b408210dce4fc639de8517c730f4995d7a6e"} Oct 29 23:04:33.209512 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.209502 5073 scope.go:122] "RemoveContainer" containerID="a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605" Oct 29 23:04:33.221194 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.221171 5073 scope.go:122] "RemoveContainer" containerID="a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605" Oct 29 23:04:33.221508 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:04:33.221487 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605\": container with ID starting with a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605 not found: ID does not exist" containerID="a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605" Oct 29 23:04:33.221582 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.221521 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605"} err="failed to get container status \"a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605\": rpc error: code = NotFound desc = could not find container \"a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605\": container with ID starting with a00cbc52fda34c6e5655b36062a8325dba2c81508a82767681932a1dbe2fa605 not found: ID does not exist" Oct 29 23:04:33.266463 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.266419 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx"] Oct 29 23:04:33.273873 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:33.273847 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-cmqlx"] Oct 29 23:04:34.838964 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:34.838927 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" path="/var/lib/kubelet/pods/256cffb1-0fc4-42f0-8496-4cd71c3a84c9/volumes" Oct 29 23:04:36.791344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:36.791299 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:04:39.256854 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:04:39.256808 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:04:45.902876 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:45.902840 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:04:45.903448 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:45.902922 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:04:45.908182 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:45.908143 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:04:45.908307 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:45.908226 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:04:57.359468 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.358722 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h"] Oct 29 23:04:57.359468 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.359400 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" podUID="4c13e2c9-7328-460a-87b1-444a9e8be8be" containerName="maas-api" containerID="cri-o://ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60" gracePeriod=30 Oct 29 23:04:57.859165 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.859121 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 23:04:57.996962 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.996923 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l\" (UniqueName: \"kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l\") pod \"4c13e2c9-7328-460a-87b1-444a9e8be8be\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " Oct 29 23:04:57.997189 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.997105 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs\") pod \"4c13e2c9-7328-460a-87b1-444a9e8be8be\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " Oct 29 23:04:57.997264 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.997192 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca\") pod \"4c13e2c9-7328-460a-87b1-444a9e8be8be\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " Oct 29 23:04:57.997264 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.997239 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls\") pod \"4c13e2c9-7328-460a-87b1-444a9e8be8be\" (UID: \"4c13e2c9-7328-460a-87b1-444a9e8be8be\") " Oct 29 23:04:57.997604 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.997573 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca" pod "4c13e2c9-7328-460a-87b1-444a9e8be8be" (UID: "4c13e2c9-7328-460a-87b1-444a9e8be8be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:57.997886 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.997864 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c13e2c9-7328-460a-87b1-444a9e8be8be-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:57.999633 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.999601 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls" pod "4c13e2c9-7328-460a-87b1-444a9e8be8be" (UID: "4c13e2c9-7328-460a-87b1-444a9e8be8be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:57.999940 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.999920 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l" pod "4c13e2c9-7328-460a-87b1-444a9e8be8be" (UID: "4c13e2c9-7328-460a-87b1-444a9e8be8be"). InnerVolumeSpecName "kube-api-access-r4z6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:57.999940 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:57.999927 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs" pod "4c13e2c9-7328-460a-87b1-444a9e8be8be" (UID: "4c13e2c9-7328-460a-87b1-444a9e8be8be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:58.099104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.099017 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.099104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.099051 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c13e2c9-7328-460a-87b1-444a9e8be8be-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.099104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.099064 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4z6l\" (UniqueName: \"kubernetes.io/projected/4c13e2c9-7328-460a-87b1-444a9e8be8be-kube-api-access-r4z6l\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.401641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.401489 5073 generic.go:356] "Generic (PLEG): container finished" podID="4c13e2c9-7328-460a-87b1-444a9e8be8be" containerID="ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60" exitCode=0 Oct 29 23:04:58.401641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.401540 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" Oct 29 23:04:58.401641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.401569 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" event={"ID":"4c13e2c9-7328-460a-87b1-444a9e8be8be","Type":"ContainerDied","Data":"ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60"} Oct 29 23:04:58.401641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.401632 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h" event={"ID":"4c13e2c9-7328-460a-87b1-444a9e8be8be","Type":"ContainerDied","Data":"77a15dc13edc0466e5cf60cae0d375253181a706e053a5b627d74e0b4f949b8d"} Oct 29 23:04:58.402301 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.401656 5073 scope.go:122] "RemoveContainer" containerID="ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60" Oct 29 23:04:58.413297 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.413275 5073 scope.go:122] "RemoveContainer" containerID="ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60" Oct 29 23:04:58.413642 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:04:58.413613 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60\": container with ID starting with ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60 not found: ID does not exist" containerID="ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60" Oct 29 23:04:58.413751 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.413656 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60"} err="failed to get container status \"ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60\": rpc error: code = NotFound desc = could not find container \"ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60\": container with ID starting with ceb24a4737fd94bb7e370c241c507e0c64c51491620e30fabadf5039e4775d60 not found: ID does not exist" Oct 29 23:04:58.456526 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.456485 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h"] Oct 29 23:04:58.469602 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.469562 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dbb507-86549b4787-smr4h"] Oct 29 23:04:58.838550 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:04:58.838519 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c13e2c9-7328-460a-87b1-444a9e8be8be" path="/var/lib/kubelet/pods/4c13e2c9-7328-460a-87b1-444a9e8be8be/volumes" Oct 29 23:05:05.158399 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:05.158341 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp"] Oct 29 23:05:05.159334 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:05.159276 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" containerID="cri-o://b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a" gracePeriod=30 Oct 29 23:05:06.311668 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:06.311636 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:05:08.027982 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:08.027946 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7"] Oct 29 23:05:08.040907 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:08.040878 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04dbbc3-796w7"] Oct 29 23:05:08.846580 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:08.846536 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4eb895b-52a2-419d-9067-5556244daebe" path="/var/lib/kubelet/pods/c4eb895b-52a2-419d-9067-5556244daebe/volumes" Oct 29 23:05:09.039297 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:09.039267 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272"] Oct 29 23:05:09.053830 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:09.053800 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7187784e-rn272"] Oct 29 23:05:09.279322 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:09.279281 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:05:10.442580 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.442520 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 23:05:10.481198 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481170 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481334 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481312 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481404 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481347 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481404 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481384 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481431 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481434 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.481517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481470 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481531 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481570 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481624 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data\") pod \"306897f3-fa2f-4117-9b46-daaf3045dfe9\" (UID: \"306897f3-fa2f-4117-9b46-daaf3045dfe9\") " Oct 29 23:05:10.481685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481625 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.481901 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.481725 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.482120 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.482052 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.482120 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.482068 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.482120 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.482091 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.482120 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.482092 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:10.482120 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.482104 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.483824 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.483803 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "kube-api-access-kxts6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:10.484256 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.484235 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:10.484350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.484272 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:10.484350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.484267 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy" pod "306897f3-fa2f-4117-9b46-daaf3045dfe9" (UID: "306897f3-fa2f-4117-9b46-daaf3045dfe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.500035 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.499610 5073 generic.go:356] "Generic (PLEG): container finished" podID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerID="b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a" exitCode=0 Oct 29 23:05:10.500035 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.499646 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" event={"ID":"306897f3-fa2f-4117-9b46-daaf3045dfe9","Type":"ContainerDied","Data":"b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a"} Oct 29 23:05:10.500035 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.499676 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" Oct 29 23:05:10.500035 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.499710 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp" event={"ID":"306897f3-fa2f-4117-9b46-daaf3045dfe9","Type":"ContainerDied","Data":"99e9d11a3c996b66a2ca4c0eefea753395a169c06f52769de5ac099e0ca64197"} Oct 29 23:05:10.500035 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.499738 5073 scope.go:122] "RemoveContainer" containerID="b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a" Oct 29 23:05:10.513809 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.513780 5073 scope.go:122] "RemoveContainer" containerID="b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a" Oct 29 23:05:10.514195 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:10.514174 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a\": container with ID starting with b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a not found: ID does not exist" containerID="b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a" Oct 29 23:05:10.514291 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.514209 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a"} err="failed to get container status \"b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a\": rpc error: code = NotFound desc = could not find container \"b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a\": container with ID starting with b68ae0b0363028ab7e70c97a70870b897ca16f9f49ed145499deda5d0011af5a not found: ID does not exist" Oct 29 23:05:10.562177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.562140 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp"] Oct 29 23:05:10.566311 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.566289 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-n8grp"] Oct 29 23:05:10.583798 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.583774 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.583798 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.583798 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.583915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.583807 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.583915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.583815 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306897f3-fa2f-4117-9b46-daaf3045dfe9-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.583915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.583823 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxts6\" (UniqueName: \"kubernetes.io/projected/306897f3-fa2f-4117-9b46-daaf3045dfe9-kube-api-access-kxts6\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.583915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.583831 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306897f3-fa2f-4117-9b46-daaf3045dfe9-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.640329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.640307 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s"] Oct 29 23:05:10.672890 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.672868 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb"] Oct 29 23:05:10.839562 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.839528 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" path="/var/lib/kubelet/pods/306897f3-fa2f-4117-9b46-daaf3045dfe9/volumes" Oct 29 23:05:10.839995 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:10.839980 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25a87b5-aab0-4a48-9269-ab734ade47c7" path="/var/lib/kubelet/pods/d25a87b5-aab0-4a48-9269-ab734ade47c7/volumes" Oct 29 23:05:20.029877 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:20.029843 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97"] Oct 29 23:05:20.043682 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:20.043656 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611836be-4hg97"] Oct 29 23:05:20.839522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:20.839486 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10961fa-e426-4173-aaa7-69a0444e3489" path="/var/lib/kubelet/pods/e10961fa-e426-4173-aaa7-69a0444e3489/volumes" Oct 29 23:05:25.661944 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:25.661872 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" containerID="cri-o://f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5" gracePeriod=45 Oct 29 23:05:25.694702 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:25.694668 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" containerID="cri-o://4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96" gracePeriod=45 Oct 29 23:05:26.063397 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.063369 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 23:05:26.072976 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.072952 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 23:05:26.153183 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.153102 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs\") pod \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " Oct 29 23:05:26.153496 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.153474 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv\" (UniqueName: \"kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv\") pod \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " Oct 29 23:05:26.153647 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.153631 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache\") pod \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " Oct 29 23:05:26.153794 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.153778 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm\") pod \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " Oct 29 23:05:26.153912 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.153899 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache\") pod \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " Oct 29 23:05:26.154592 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.154567 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home\") pod \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " Oct 29 23:05:26.154781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.154766 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7\" (UniqueName: \"kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7\") pod \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " Oct 29 23:05:26.154970 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.154955 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home\") pod \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " Oct 29 23:05:26.155085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.155072 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm\") pod \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " Oct 29 23:05:26.155246 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.155230 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir\") pod \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\" (UID: \"a0da1f6f-33f4-411f-806e-6e1e37fc03f6\") " Oct 29 23:05:26.155383 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.155369 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir\") pod \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " Oct 29 23:05:26.155516 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.155504 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs\") pod \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\" (UID: \"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d\") " Oct 29 23:05:26.163668 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.163211 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir" pod "a0da1f6f-33f4-411f-806e-6e1e37fc03f6" (UID: "a0da1f6f-33f4-411f-806e-6e1e37fc03f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.163668 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.163401 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir" pod "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" (UID: "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.164131 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.164097 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache" pod "a0da1f6f-33f4-411f-806e-6e1e37fc03f6" (UID: "a0da1f6f-33f4-411f-806e-6e1e37fc03f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.164549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.164530 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home" pod "a0da1f6f-33f4-411f-806e-6e1e37fc03f6" (UID: "a0da1f6f-33f4-411f-806e-6e1e37fc03f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.164847 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.164830 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache" pod "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" (UID: "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.165015 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.164998 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home" pod "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" (UID: "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.170046 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.170010 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm" pod "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" (UID: "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.170888 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.170853 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv" pod "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" (UID: "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d"). InnerVolumeSpecName "kube-api-access-5vmbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:26.171715 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.171653 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs" pod "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" (UID: "c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:26.173245 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.172113 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7" pod "a0da1f6f-33f4-411f-806e-6e1e37fc03f6" (UID: "a0da1f6f-33f4-411f-806e-6e1e37fc03f6"). InnerVolumeSpecName "kube-api-access-zktg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:26.173245 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.172728 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm" pod "a0da1f6f-33f4-411f-806e-6e1e37fc03f6" (UID: "a0da1f6f-33f4-411f-806e-6e1e37fc03f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.174247 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.174225 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs" pod "a0da1f6f-33f4-411f-806e-6e1e37fc03f6" (UID: "a0da1f6f-33f4-411f-806e-6e1e37fc03f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257301 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257333 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257343 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257351 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257360 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257368 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257377 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmbv\" (UniqueName: \"kubernetes.io/projected/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-kube-api-access-5vmbv\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257386 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257394 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257402 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257409 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.257755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.257416 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zktg7\" (UniqueName: \"kubernetes.io/projected/a0da1f6f-33f4-411f-806e-6e1e37fc03f6-kube-api-access-zktg7\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.608830 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.608739 5073 generic.go:356] "Generic (PLEG): container finished" podID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerID="4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96" exitCode=0 Oct 29 23:05:26.608830 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.608785 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" Oct 29 23:05:26.609018 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.608821 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" event={"ID":"a0da1f6f-33f4-411f-806e-6e1e37fc03f6","Type":"ContainerDied","Data":"4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96"} Oct 29 23:05:26.609018 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.608869 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb" event={"ID":"a0da1f6f-33f4-411f-806e-6e1e37fc03f6","Type":"ContainerDied","Data":"1a2f0f4f929d58c5b30f91f9407a4febf15bf5a26360620d72bf9ab12a420e22"} Oct 29 23:05:26.609018 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.608887 5073 scope.go:122] "RemoveContainer" containerID="4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96" Oct 29 23:05:26.610611 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.610548 5073 generic.go:356] "Generic (PLEG): container finished" podID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerID="f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5" exitCode=0 Oct 29 23:05:26.610611 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.610583 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" Oct 29 23:05:26.610611 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.610597 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" event={"ID":"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d","Type":"ContainerDied","Data":"f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5"} Oct 29 23:05:26.610611 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.610636 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s" event={"ID":"c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d","Type":"ContainerDied","Data":"ac629943d79ea2b873cc6e47043ed9e0c53c0ea467f644632fcea05235de47bb"} Oct 29 23:05:26.619322 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.619301 5073 scope.go:122] "RemoveContainer" containerID="4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96" Oct 29 23:05:26.619611 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:26.619593 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96\": container with ID starting with 4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96 not found: ID does not exist" containerID="4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96" Oct 29 23:05:26.619688 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.619620 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96"} err="failed to get container status \"4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96\": rpc error: code = NotFound desc = could not find container \"4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96\": container with ID starting with 4c3173b86c32988487d8a278629d2b34789c3928567cf1b8f1bd54d787d94f96 not found: ID does not exist" Oct 29 23:05:26.619688 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.619637 5073 scope.go:122] "RemoveContainer" containerID="f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5" Oct 29 23:05:26.663998 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.663757 5073 scope.go:122] "RemoveContainer" containerID="f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5" Oct 29 23:05:26.664341 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:26.664114 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5\": container with ID starting with f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5 not found: ID does not exist" containerID="f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5" Oct 29 23:05:26.664341 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.664142 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5"} err="failed to get container status \"f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5\": rpc error: code = NotFound desc = could not find container \"f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5\": container with ID starting with f8f676d82c0aa4d084622aff795f1703b0339b107c59fd3ce851bebd0e1921c5 not found: ID does not exist" Oct 29 23:05:26.675283 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.675214 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb"] Oct 29 23:05:26.688484 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.688453 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/simulator-w5-0d12c5-kserve-585b9fcf57-872pb"] Oct 29 23:05:26.723723 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.723697 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s"] Oct 29 23:05:26.731706 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.731683 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/distinct-w5-0d12c5-kserve-86b7c56cff-b242s"] Oct 29 23:05:26.838585 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.838558 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" path="/var/lib/kubelet/pods/a0da1f6f-33f4-411f-806e-6e1e37fc03f6/volumes" Oct 29 23:05:26.839078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:26.839056 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" path="/var/lib/kubelet/pods/c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d/volumes" Oct 29 23:05:35.928172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:35.927490 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn"] Oct 29 23:05:35.928172 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:35.927746 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" containerID="cri-o://69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78" gracePeriod=30 Oct 29 23:05:36.607596 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:36.607558 5073 patch_prober.go:29] interesting pod/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:05:36.607774 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:36.607652 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:05:38.636870 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:38.636836 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:05:39.255623 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:39.255584 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:05:41.205587 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.205561 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:05:41.306210 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306174 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306381 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306246 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306381 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306280 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306381 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306367 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306586 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306420 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306586 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306461 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306586 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306519 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306586 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306578 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306808 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306634 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket\") pod \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\" (UID: \"e64aaf8d-64fe-4420-bdb5-a604c82f4305\") " Oct 29 23:05:41.306808 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.306688 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:41.307299 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.307267 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:41.307299 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.307284 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:41.307482 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.307358 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:41.307482 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.307407 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.307607 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.307534 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:41.309032 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.308992 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "kube-api-access-szj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:41.309185 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.309137 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:41.309340 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.309318 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:41.309686 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.309659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token" pod "e64aaf8d-64fe-4420-bdb5-a604c82f4305" (UID: "e64aaf8d-64fe-4420-bdb5-a604c82f4305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:41.408318 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408290 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408318 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408314 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408323 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szj7g\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-kube-api-access-szj7g\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408333 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408341 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408350 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408357 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64aaf8d-64fe-4420-bdb5-a604c82f4305-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.408457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.408365 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64aaf8d-64fe-4420-bdb5-a604c82f4305-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:41.714265 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.714234 5073 generic.go:356] "Generic (PLEG): container finished" podID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerID="69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78" exitCode=0 Oct 29 23:05:41.714424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.714285 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" Oct 29 23:05:41.714424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.714297 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" event={"ID":"e64aaf8d-64fe-4420-bdb5-a604c82f4305","Type":"ContainerDied","Data":"69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78"} Oct 29 23:05:41.714424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.714385 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn" event={"ID":"e64aaf8d-64fe-4420-bdb5-a604c82f4305","Type":"ContainerDied","Data":"ff920fc1da3737804edb4a7d0d45dee016046b690f6fd1c4afa9d4a3b617474f"} Oct 29 23:05:41.714424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.714415 5073 scope.go:122] "RemoveContainer" containerID="69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78" Oct 29 23:05:41.725075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.725057 5073 scope.go:122] "RemoveContainer" containerID="aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea" Oct 29 23:05:41.734709 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.734689 5073 scope.go:122] "RemoveContainer" containerID="69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78" Oct 29 23:05:41.734985 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:41.734965 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78\": container with ID starting with 69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78 not found: ID does not exist" containerID="69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78" Oct 29 23:05:41.735050 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.734994 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78"} err="failed to get container status \"69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78\": rpc error: code = NotFound desc = could not find container \"69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78\": container with ID starting with 69268b6c56585f174e8e130f34b1cecfa69d43251b4692d99a3d7944545a2b78 not found: ID does not exist" Oct 29 23:05:41.735050 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.735012 5073 scope.go:122] "RemoveContainer" containerID="aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea" Oct 29 23:05:41.735273 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:05:41.735253 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea\": container with ID starting with aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea not found: ID does not exist" containerID="aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea" Oct 29 23:05:41.735342 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.735283 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea"} err="failed to get container status \"aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea\": rpc error: code = NotFound desc = could not find container \"aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea\": container with ID starting with aeb7b0606dc0799ca91128e4db1e3369956adcb99e31e74bfb7541bce6a732ea not found: ID does not exist" Oct 29 23:05:41.785703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.785678 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn"] Oct 29 23:05:41.798919 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:41.798900 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3e2b2c-openshift-default-54764b7f65-m6nvn"] Oct 29 23:05:42.837949 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:42.837903 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" path="/var/lib/kubelet/pods/e64aaf8d-64fe-4420-bdb5-a604c82f4305/volumes" Oct 29 23:05:52.568361 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:52.568321 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc"] Oct 29 23:05:52.568876 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:52.568565 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" containerID="cri-o://22a42d5c5fb57c41a02a13fa17c78fc6f859a0639a9060ab55b0b094d15da3cc" gracePeriod=30 Oct 29 23:05:57.540227 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.540190 5073 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:05:57.540722 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.540255 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:05:57.838448 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.838362 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerID="22a42d5c5fb57c41a02a13fa17c78fc6f859a0639a9060ab55b0b094d15da3cc" exitCode=0 Oct 29 23:05:57.838448 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.838431 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" event={"ID":"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0","Type":"ContainerDied","Data":"22a42d5c5fb57c41a02a13fa17c78fc6f859a0639a9060ab55b0b094d15da3cc"} Oct 29 23:05:57.872354 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.872329 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 23:05:57.930329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930295 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930545 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930417 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930545 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930489 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930990 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930547 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930990 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930584 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930990 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930614 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930990 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930663 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930990 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930734 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.930990 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.930783 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket\") pod \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\" (UID: \"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0\") " Oct 29 23:05:57.932021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.931593 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.932021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.931977 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.932021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.931992 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:57.932286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.932062 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.932286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.932081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:57.938241 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.934476 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:57.938241 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.934966 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:57.938241 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.936588 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "kube-api-access-5776t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:57.941407 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:57.938816 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy" pod "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" (UID: "b2d3d4bf-809e-4567-ad16-1b0f14a04ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:58.032291 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032263 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5776t\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-kube-api-access-5776t\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032291 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032290 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032300 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032308 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032315 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032323 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032331 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032339 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.032435 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.032346 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.845500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.845464 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" Oct 29 23:05:58.845500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.845479 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc" event={"ID":"b2d3d4bf-809e-4567-ad16-1b0f14a04ca0","Type":"ContainerDied","Data":"ba2bf53aa158f2359ed7ca1995f16fbdef926e9f2d763aa96dbb9103aaf5e7c7"} Oct 29 23:05:58.846036 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.845529 5073 scope.go:122] "RemoveContainer" containerID="22a42d5c5fb57c41a02a13fa17c78fc6f859a0639a9060ab55b0b094d15da3cc" Oct 29 23:05:58.916741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.916705 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc"] Oct 29 23:05:58.927862 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:05:58.927832 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-5hwzc"] Oct 29 23:06:00.838265 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:00.838235 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" path="/var/lib/kubelet/pods/b2d3d4bf-809e-4567-ad16-1b0f14a04ca0/volumes" Oct 29 23:06:08.547604 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:08.547571 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:06:09.254863 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:09.254829 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:06:18.221637 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.221603 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh"] Oct 29 23:06:18.222124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.221810 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" podUID="07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" containerName="maas-api" containerID="cri-o://1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae" gracePeriod=30 Oct 29 23:06:18.501118 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.501085 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 23:06:18.504198 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.504171 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb\" (UniqueName: \"kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb\") pod \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " Oct 29 23:06:18.504311 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.504228 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs\") pod \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " Oct 29 23:06:18.504311 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.504280 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca\") pod \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " Oct 29 23:06:18.504388 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.504323 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls\") pod \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\" (UID: \"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3\") " Oct 29 23:06:18.504641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.504613 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca" pod "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" (UID: "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:18.506430 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.506399 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb" pod "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" (UID: "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3"). InnerVolumeSpecName "kube-api-access-67lxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:18.506536 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.506444 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls" pod "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" (UID: "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:18.506584 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.506566 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs" pod "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" (UID: "07ed4fe7-ea1e-4f13-9bf1-f494f67459b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:18.606168 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.606045 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lxb\" (UniqueName: \"kubernetes.io/projected/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-kube-api-access-67lxb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:18.606168 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.606081 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:18.606168 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.606092 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:18.606168 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.606101 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:18.989119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.989083 5073 generic.go:356] "Generic (PLEG): container finished" podID="07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" containerID="1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae" exitCode=0 Oct 29 23:06:18.989325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.989126 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" Oct 29 23:06:18.989325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.989187 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" event={"ID":"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3","Type":"ContainerDied","Data":"1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae"} Oct 29 23:06:18.989325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.989234 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh" event={"ID":"07ed4fe7-ea1e-4f13-9bf1-f494f67459b3","Type":"ContainerDied","Data":"32a6f49bae1d54d993a92ebd9bad9ceb8e07968e47dfa1f309268e5e15034ef6"} Oct 29 23:06:18.989325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:18.989258 5073 scope.go:122] "RemoveContainer" containerID="1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae" Oct 29 23:06:19.000427 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:19.000408 5073 scope.go:122] "RemoveContainer" containerID="1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae" Oct 29 23:06:19.000732 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:19.000712 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae\": container with ID starting with 1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae not found: ID does not exist" containerID="1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae" Oct 29 23:06:19.000805 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:19.000742 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae"} err="failed to get container status \"1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae\": rpc error: code = NotFound desc = could not find container \"1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae\": container with ID starting with 1ca5dbb332391fef7b57fb0e6092ff145cf8335442f37fff720b62c0ea32a6ae not found: ID does not exist" Oct 29 23:06:19.042645 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:19.042616 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh"] Oct 29 23:06:19.055687 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:19.055663 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6bb06a-7c697f76fc-stpsh"] Oct 29 23:06:20.845881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:20.845838 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" path="/var/lib/kubelet/pods/07ed4fe7-ea1e-4f13-9bf1-f494f67459b3/volumes" Oct 29 23:06:23.560793 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:23.560759 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:06:23.898347 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:23.898270 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9"] Oct 29 23:06:23.898505 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:23.898483 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" containerID="cri-o://0d904883d64207714db50ccb9ff816abb37e25c75ad70258eb3f2d66fc746788" gracePeriod=30 Oct 29 23:06:26.138929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:26.138889 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn"] Oct 29 23:06:26.139468 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:26.139135 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" containerID="cri-o://1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b" gracePeriod=30 Oct 29 23:06:29.061102 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.061067 5073 generic.go:356] "Generic (PLEG): container finished" podID="d04ef323-4e21-490d-a784-4e71f002b17a" containerID="0d904883d64207714db50ccb9ff816abb37e25c75ad70258eb3f2d66fc746788" exitCode=0 Oct 29 23:06:29.061517 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.061132 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" event={"ID":"d04ef323-4e21-490d-a784-4e71f002b17a","Type":"ContainerDied","Data":"0d904883d64207714db50ccb9ff816abb37e25c75ad70258eb3f2d66fc746788"} Oct 29 23:06:29.164600 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.164573 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 23:06:29.257842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.257815 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258031 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.257863 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258031 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258004 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258045 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258090 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258310 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258170 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258310 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258216 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258310 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258279 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258470 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258348 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs\") pod \"d04ef323-4e21-490d-a784-4e71f002b17a\" (UID: \"d04ef323-4e21-490d-a784-4e71f002b17a\") " Oct 29 23:06:29.258939 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258275 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:29.258939 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258544 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.258939 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.258725 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.259217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.259002 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.259217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.259023 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.259217 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.259035 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ef323-4e21-490d-a784-4e71f002b17a-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.259394 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.259235 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.259394 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.259306 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.260701 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.260674 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:29.260954 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.260922 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:29.261074 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.261011 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.261269 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.261251 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t" pod "d04ef323-4e21-490d-a784-4e71f002b17a" (UID: "d04ef323-4e21-490d-a784-4e71f002b17a"). InnerVolumeSpecName "kube-api-access-n8b8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:29.359832 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.359803 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8b8t\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-kube-api-access-n8b8t\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.359832 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.359832 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ef323-4e21-490d-a784-4e71f002b17a-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.359961 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.359841 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.359961 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.359851 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.359961 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.359859 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ef323-4e21-490d-a784-4e71f002b17a-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.359961 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:29.359865 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ef323-4e21-490d-a784-4e71f002b17a-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:30.068673 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:30.068638 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" Oct 29 23:06:30.069126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:30.068719 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9" event={"ID":"d04ef323-4e21-490d-a784-4e71f002b17a","Type":"ContainerDied","Data":"ee9a155028841bd1f555a604e6f570e7497754de2dd31db1a3562cc02d246e7a"} Oct 29 23:06:30.069126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:30.068786 5073 scope.go:122] "RemoveContainer" containerID="0d904883d64207714db50ccb9ff816abb37e25c75ad70258eb3f2d66fc746788" Oct 29 23:06:30.141092 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:30.141071 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9"] Oct 29 23:06:30.151893 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:30.151871 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-pgpr9"] Oct 29 23:06:30.838082 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:30.838047 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" path="/var/lib/kubelet/pods/d04ef323-4e21-490d-a784-4e71f002b17a/volumes" Oct 29 23:06:31.178275 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.178248 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 23:06:31.178589 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.178297 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 23:06:31.410794 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.410734 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 23:06:31.481547 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481509 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.481703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.481703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481607 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.481703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481656 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.481892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481789 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.481892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481806 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.481892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481832 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.481892 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481868 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.482186 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481894 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.482186 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481930 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket\") pod \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\" (UID: \"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227\") " Oct 29 23:06:31.482186 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.481928 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.482417 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.482611 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.482656 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.482671 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.482679 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.484634 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.484955 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.484989 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:31.485906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.485021 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.487098 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.487071 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc" pod "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" (UID: "cd886ec9-e3ab-4b33-bf0b-3a574bcdf227"). InnerVolumeSpecName "kube-api-access-5hnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.584016 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.583991 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.584016 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.584015 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnpc\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-kube-api-access-5hnpc\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.584131 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.584032 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.584131 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.584041 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.584131 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.584049 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.584131 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.584056 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.622109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.622087 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj"] Oct 29 23:06:31.630731 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:31.630702 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl"] Oct 29 23:06:32.083523 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.083480 5073 generic.go:356] "Generic (PLEG): container finished" podID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerID="1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b" exitCode=0 Oct 29 23:06:32.083719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.083562 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" Oct 29 23:06:32.083719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.083574 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" event={"ID":"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227","Type":"ContainerDied","Data":"1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b"} Oct 29 23:06:32.083719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.083652 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn" event={"ID":"cd886ec9-e3ab-4b33-bf0b-3a574bcdf227","Type":"ContainerDied","Data":"da2d0b73b1f6617f2aa4dce2cbaa1cbaca41869443a0fc6ae691d88236ee75d6"} Oct 29 23:06:32.083719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.083678 5073 scope.go:122] "RemoveContainer" containerID="1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b" Oct 29 23:06:32.094477 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.094459 5073 scope.go:122] "RemoveContainer" containerID="3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2" Oct 29 23:06:32.103711 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.103692 5073 scope.go:122] "RemoveContainer" containerID="1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b" Oct 29 23:06:32.103959 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:32.103943 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b\": container with ID starting with 1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b not found: ID does not exist" containerID="1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b" Oct 29 23:06:32.104014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.103966 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b"} err="failed to get container status \"1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b\": rpc error: code = NotFound desc = could not find container \"1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b\": container with ID starting with 1987adb343ab9741360403d94ba19c1078054d567ff7d31b5259625c4def097b not found: ID does not exist" Oct 29 23:06:32.104014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.103980 5073 scope.go:122] "RemoveContainer" containerID="3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2" Oct 29 23:06:32.104280 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:32.104261 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2\": container with ID starting with 3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2 not found: ID does not exist" containerID="3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2" Oct 29 23:06:32.104341 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.104289 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2"} err="failed to get container status \"3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2\": rpc error: code = NotFound desc = could not find container \"3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2\": container with ID starting with 3e9c2dd548031c2a68cc22aeb7a56ab6baf348faf4ee864b48bf736e0d1216f2 not found: ID does not exist" Oct 29 23:06:32.150307 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.150276 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn"] Oct 29 23:06:32.164282 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.164256 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-smqnn"] Oct 29 23:06:32.838624 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:32.838590 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" path="/var/lib/kubelet/pods/cd886ec9-e3ab-4b33-bf0b-3a574bcdf227/volumes" Oct 29 23:06:35.616654 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:35.616618 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6"] Oct 29 23:06:35.617142 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:35.616813 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" containerID="cri-o://9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba" gracePeriod=30 Oct 29 23:06:37.040966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:37.040928 5073 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:37.041442 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:37.040997 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:38.040710 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.040668 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7"] Oct 29 23:06:38.040962 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.040916 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" podUID="7c0fa98a-d57b-414d-b37b-1534ac637df5" containerName="maas-api" containerID="cri-o://38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614" gracePeriod=30 Oct 29 23:06:38.376000 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.375973 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 23:06:38.423176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.413469 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs\") pod \"7c0fa98a-d57b-414d-b37b-1534ac637df5\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " Oct 29 23:06:38.423176 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.413601 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls\") pod \"7c0fa98a-d57b-414d-b37b-1534ac637df5\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " Oct 29 23:06:38.423438 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.423189 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz\" (UniqueName: \"kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz\") pod \"7c0fa98a-d57b-414d-b37b-1534ac637df5\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " Oct 29 23:06:38.423438 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.423249 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca\") pod \"7c0fa98a-d57b-414d-b37b-1534ac637df5\" (UID: \"7c0fa98a-d57b-414d-b37b-1534ac637df5\") " Oct 29 23:06:38.431839 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.426679 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca" pod "7c0fa98a-d57b-414d-b37b-1534ac637df5" (UID: "7c0fa98a-d57b-414d-b37b-1534ac637df5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:38.441175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.433127 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz" pod "7c0fa98a-d57b-414d-b37b-1534ac637df5" (UID: "7c0fa98a-d57b-414d-b37b-1534ac637df5"). InnerVolumeSpecName "kube-api-access-dpgqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:38.454422 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.448547 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls" pod "7c0fa98a-d57b-414d-b37b-1534ac637df5" (UID: "7c0fa98a-d57b-414d-b37b-1534ac637df5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:38.458728 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.458318 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs" pod "7c0fa98a-d57b-414d-b37b-1534ac637df5" (UID: "7c0fa98a-d57b-414d-b37b-1534ac637df5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:38.468823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.468784 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn"] Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469538 5073 cpu_manager.go:414] "containerMap: removing container" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469569 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469588 5073 cpu_manager.go:414] "containerMap: removing container" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469596 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469608 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469615 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469629 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469636 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469653 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469662 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469673 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4c13e2c9-7328-460a-87b1-444a9e8be8be" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469680 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c13e2c9-7328-460a-87b1-444a9e8be8be" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469694 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" containerName="payload-processing" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469700 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" containerName="payload-processing" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469711 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2cc50559-210a-4f52-af82-4f2b78a28bcb" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469717 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc50559-210a-4f52-af82-4f2b78a28bcb" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469736 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469743 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469754 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469761 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469774 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469780 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469795 5073 cpu_manager.go:414] "containerMap: removing container" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469802 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469810 5073 cpu_manager.go:414] "containerMap: removing container" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469816 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469826 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7c0fa98a-d57b-414d-b37b-1534ac637df5" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469833 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0fa98a-d57b-414d-b37b-1534ac637df5" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469847 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469853 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469864 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469871 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469887 5073 cpu_manager.go:414] "containerMap: removing container" podUID="93100a9f-ba38-4855-99a4-527321296c7b" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469894 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93100a9f-ba38-4855-99a4-527321296c7b" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469906 5073 cpu_manager.go:414] "containerMap: removing container" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469913 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469939 5073 cpu_manager.go:414] "containerMap: removing container" podUID="972766dc-cfeb-425c-ab30-64188633483b" containerName="payload-pre-processing" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469947 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972766dc-cfeb-425c-ab30-64188633483b" containerName="payload-pre-processing" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469961 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469968 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469979 5073 cpu_manager.go:414] "containerMap: removing container" podUID="07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.469986 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470001 5073 cpu_manager.go:414] "containerMap: removing container" podUID="20b62beb-82c4-4577-bf93-cc54642f96f9" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470008 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b62beb-82c4-4577-bf93-cc54642f96f9" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470017 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470024 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470035 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470042 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470058 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470065 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470074 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f1402654-ca43-41d9-afd6-94f7966c9783" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470082 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1402654-ca43-41d9-afd6-94f7966c9783" containerName="maas-api" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470093 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470100 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470116 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" Oct 29 23:06:38.472284 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.470122 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472349 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0da1f6f-33f4-411f-806e-6e1e37fc03f6" containerName="main" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472381 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d3d4bf-809e-4567-ad16-1b0f14a04ca0" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472393 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1402654-ca43-41d9-afd6-94f7966c9783" containerName="maas-api" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472402 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f8dde8-ebda-49cd-b14a-825bea2f48cd" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472413 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04ef323-4e21-490d-a784-4e71f002b17a" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472422 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="053148a5-5ce2-4131-9aab-c09455de26b4" containerName="main" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472430 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="66145ed4-2278-4025-ac12-639607aeae8f" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472442 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472450 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec36d33-ee32-4d02-b608-3170bff96d4b" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472461 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8ffa39-241f-4f94-9d5b-4ecdbecee27a" containerName="main" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472468 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c13e2c9-7328-460a-87b1-444a9e8be8be" containerName="maas-api" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472479 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5c6ff7-ca44-45c6-bc98-c27a3a24f54e" containerName="payload-processing" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472487 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc50559-210a-4f52-af82-4f2b78a28bcb" containerName="maas-api" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472497 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c2e6ae-8ac3-4286-8dad-d44e63acdb4d" containerName="main" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472506 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd886ec9-e3ab-4b33-bf0b-3a574bcdf227" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472516 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472526 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="256cffb1-0fc4-42f0-8496-4cd71c3a84c9" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472535 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="306897f3-fa2f-4117-9b46-daaf3045dfe9" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472543 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0fa98a-d57b-414d-b37b-1534ac637df5" containerName="maas-api" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472553 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64aaf8d-64fe-4420-bdb5-a604c82f4305" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472561 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8862115c-5cd4-4636-bba6-16d04553feb9" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472569 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="93100a9f-ba38-4855-99a4-527321296c7b" containerName="main" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472579 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca898de-4510-4f38-b7f9-bfb6a32cb466" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472590 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="972766dc-cfeb-425c-ab30-64188633483b" containerName="payload-pre-processing" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472599 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bf2ed8-0ed0-4b75-9fa1-6dab7cc3ee0c" containerName="istio-proxy" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472608 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ed4fe7-ea1e-4f13-9bf1-f494f67459b3" containerName="maas-api" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472619 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b62beb-82c4-4577-bf93-cc54642f96f9" containerName="maas-api" Oct 29 23:06:38.474823 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.472629 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddf42fd-58e3-46a1-9f57-8bebecf625a5" containerName="istio-proxy" Oct 29 23:06:38.476680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.476657 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.492324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.492301 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:38.492495 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.492475 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:06:38.494326 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.494303 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn"] Oct 29 23:06:38.525655 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.525628 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.525776 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.525743 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.525829 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.525792 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fj6j\" (UniqueName: \"kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.525939 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.525926 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.526001 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.525978 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0fa98a-d57b-414d-b37b-1534ac637df5-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.526052 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.526001 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpgqz\" (UniqueName: \"kubernetes.io/projected/7c0fa98a-d57b-414d-b37b-1534ac637df5-kube-api-access-dpgqz\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.526052 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.526019 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0fa98a-d57b-414d-b37b-1534ac637df5-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.627373 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.627295 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.627373 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.627343 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fj6j\" (UniqueName: \"kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.627605 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.627477 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.628136 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.628106 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.629958 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.629929 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.648286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.648251 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fj6j\" (UniqueName: \"kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j\") pod \"maas-api-revoke-sub-249ad63e-nbbkn\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:38.790656 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:38.790629 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:39.095941 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.095909 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn"] Oct 29 23:06:39.097475 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:06:39.097447 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4278c6_0bad_4065_b53a_5b886753d055.slice/crio-eccceb4433075b13a4c1d2318bcc45141901c8642e2094a31893f2f3e9f7675f WatchSource:0}: Error finding container eccceb4433075b13a4c1d2318bcc45141901c8642e2094a31893f2f3e9f7675f: Status 404 returned error can't find the container with id eccceb4433075b13a4c1d2318bcc45141901c8642e2094a31893f2f3e9f7675f Oct 29 23:06:39.129746 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.129719 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c0fa98a-d57b-414d-b37b-1534ac637df5" containerID="38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614" exitCode=0 Oct 29 23:06:39.129862 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.129766 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" Oct 29 23:06:39.129862 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.129797 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" event={"ID":"7c0fa98a-d57b-414d-b37b-1534ac637df5","Type":"ContainerDied","Data":"38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614"} Oct 29 23:06:39.129862 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.129847 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7" event={"ID":"7c0fa98a-d57b-414d-b37b-1534ac637df5","Type":"ContainerDied","Data":"fb374c3d1b7b28920fa9c439c1ef49af1457113e262a6381b93b0f6c751c3776"} Oct 29 23:06:39.130051 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.129878 5073 scope.go:122] "RemoveContainer" containerID="38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614" Oct 29 23:06:39.135447 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.135345 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" event={"ID":"5a4278c6-0bad-4065-b53a-5b886753d055","Type":"ContainerStarted","Data":"eccceb4433075b13a4c1d2318bcc45141901c8642e2094a31893f2f3e9f7675f"} Oct 29 23:06:39.143337 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.143316 5073 scope.go:122] "RemoveContainer" containerID="38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614" Oct 29 23:06:39.143625 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:39.143600 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614\": container with ID starting with 38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614 not found: ID does not exist" containerID="38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614" Oct 29 23:06:39.143716 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.143635 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614"} err="failed to get container status \"38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614\": rpc error: code = NotFound desc = could not find container \"38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614\": container with ID starting with 38937cad2dc6361c22dd32d4ba3827a60b7191b4ff4c2a431082f2495ada6614 not found: ID does not exist" Oct 29 23:06:39.185338 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.185315 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7"] Oct 29 23:06:39.198238 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:39.198217 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2ae845-6bb4dc7cc9-zktq7"] Oct 29 23:06:39.253441 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:39.253414 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:06:40.144157 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.144124 5073 generic.go:356] "Generic (PLEG): container finished" podID="5a4278c6-0bad-4065-b53a-5b886753d055" containerID="a96cbe9c89bb6ad9600a26ad895c26d5bd03568d605bc2269a8bfd5cad7c8878" exitCode=0 Oct 29 23:06:40.144513 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.144204 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" event={"ID":"5a4278c6-0bad-4065-b53a-5b886753d055","Type":"ContainerDied","Data":"a96cbe9c89bb6ad9600a26ad895c26d5bd03568d605bc2269a8bfd5cad7c8878"} Oct 29 23:06:40.839200 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.839145 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0fa98a-d57b-414d-b37b-1534ac637df5" path="/var/lib/kubelet/pods/7c0fa98a-d57b-414d-b37b-1534ac637df5/volumes" Oct 29 23:06:40.894044 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.893987 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:06:40.964258 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964231 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964412 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964267 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964412 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964321 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964422 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964453 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964487 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964521 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964613 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964615 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:40.964697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964692 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token\") pod \"a7448d27-6f4d-4020-9525-7f82d08a3806\" (UID: \"a7448d27-6f4d-4020-9525-7f82d08a3806\") " Oct 29 23:06:40.964915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964787 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:40.965043 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.964871 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:40.965239 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965057 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:40.965325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965249 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:40.965422 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965398 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:40.965422 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965418 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7448d27-6f4d-4020-9525-7f82d08a3806-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:40.965752 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965431 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:40.965752 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965443 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:40.965752 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.965451 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:40.967310 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.967278 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:40.967488 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.967465 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "kube-api-access-f6z6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:40.967702 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.967681 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:40.967790 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:40.967701 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy" pod "a7448d27-6f4d-4020-9525-7f82d08a3806" (UID: "a7448d27-6f4d-4020-9525-7f82d08a3806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.067124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.067094 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.067124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.067123 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.067300 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.067136 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6z6m\" (UniqueName: \"kubernetes.io/projected/a7448d27-6f4d-4020-9525-7f82d08a3806-kube-api-access-f6z6m\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.067300 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.067184 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7448d27-6f4d-4020-9525-7f82d08a3806-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.152556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.152460 5073 generic.go:356] "Generic (PLEG): container finished" podID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerID="9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba" exitCode=0 Oct 29 23:06:41.152556 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.152511 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" Oct 29 23:06:41.153344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.152837 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" event={"ID":"a7448d27-6f4d-4020-9525-7f82d08a3806","Type":"ContainerDied","Data":"9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba"} Oct 29 23:06:41.153344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.152912 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6" event={"ID":"a7448d27-6f4d-4020-9525-7f82d08a3806","Type":"ContainerDied","Data":"06bd11b8a48790e05edf6b86aa604dc9a2b0c0ca556e1b35aa654da890a67311"} Oct 29 23:06:41.153344 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.152938 5073 scope.go:122] "RemoveContainer" containerID="9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba" Oct 29 23:06:41.167729 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.167710 5073 scope.go:122] "RemoveContainer" containerID="9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba" Oct 29 23:06:41.168055 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:41.168035 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba\": container with ID starting with 9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba not found: ID does not exist" containerID="9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba" Oct 29 23:06:41.168130 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.168067 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba"} err="failed to get container status \"9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba\": rpc error: code = NotFound desc = could not find container \"9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba\": container with ID starting with 9ce23a1425f0596892678ccfede49fd4fd7a8d1025cf47ca2e9e1d55cb4d51ba not found: ID does not exist" Oct 29 23:06:41.232265 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.232232 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6"] Oct 29 23:06:41.241268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.241236 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-hwrz6"] Oct 29 23:06:41.241691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.241668 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:06:41.339474 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.339446 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:41.471048 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.471023 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca\") pod \"5a4278c6-0bad-4065-b53a-5b886753d055\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " Oct 29 23:06:41.471316 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.471295 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca\") pod \"5a4278c6-0bad-4065-b53a-5b886753d055\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " Oct 29 23:06:41.471385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.471331 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j\" (UniqueName: \"kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j\") pod \"5a4278c6-0bad-4065-b53a-5b886753d055\" (UID: \"5a4278c6-0bad-4065-b53a-5b886753d055\") " Oct 29 23:06:41.471385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.471358 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca" pod "5a4278c6-0bad-4065-b53a-5b886753d055" (UID: "5a4278c6-0bad-4065-b53a-5b886753d055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:41.471646 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.471628 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.473531 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.473504 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca" pod "5a4278c6-0bad-4065-b53a-5b886753d055" (UID: "5a4278c6-0bad-4065-b53a-5b886753d055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:41.473647 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.473568 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j" pod "5a4278c6-0bad-4065-b53a-5b886753d055" (UID: "5a4278c6-0bad-4065-b53a-5b886753d055"). InnerVolumeSpecName "kube-api-access-9fj6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.580042 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.579973 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4278c6-0bad-4065-b53a-5b886753d055-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.580042 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:41.580015 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fj6j\" (UniqueName: \"kubernetes.io/projected/5a4278c6-0bad-4065-b53a-5b886753d055-kube-api-access-9fj6j\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:42.160881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:42.160831 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" Oct 29 23:06:42.160881 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:42.160855 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn" event={"ID":"5a4278c6-0bad-4065-b53a-5b886753d055","Type":"ContainerDied","Data":"eccceb4433075b13a4c1d2318bcc45141901c8642e2094a31893f2f3e9f7675f"} Oct 29 23:06:42.161490 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:42.160907 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eccceb4433075b13a4c1d2318bcc45141901c8642e2094a31893f2f3e9f7675f" Oct 29 23:06:42.838107 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:42.838066 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" path="/var/lib/kubelet/pods/a7448d27-6f4d-4020-9525-7f82d08a3806/volumes" Oct 29 23:06:45.438537 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:45.438504 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f"] Oct 29 23:06:45.439085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:45.438772 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" containerID="cri-o://98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a" gracePeriod=30 Oct 29 23:06:46.458336 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.458300 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw"] Oct 29 23:06:46.458884 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.458566 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" containerName="discovery" containerID="cri-o://e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420" gracePeriod=30 Oct 29 23:06:46.645026 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.644476 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" containerID="cri-o://5056e022f3bc80c82830dcf9ee1c8b503add1d2843abcd89133344e3ef7029fc" gracePeriod=45 Oct 29 23:06:46.664050 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.664007 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" containerID="cri-o://03906e1f8c098c0777bd269557010aebe9ac266aa5cef02740cae6532556c41b" gracePeriod=45 Oct 29 23:06:46.675748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.675712 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused" Oct 29 23:06:46.805703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.805676 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.936759 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.936815 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.936869 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.936899 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.937041 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.937106 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.937835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.937196 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig\") pod \"78256170-0e88-45a6-84aa-f4c81d35d722\" (UID: \"78256170-0e88-45a6-84aa-f4c81d35d722\") " Oct 29 23:06:46.938328 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.937991 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:46.939899 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.939856 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:46.940878 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.940856 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:46.941477 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.941456 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:46.943510 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.943484 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "kube-api-access-dlrgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:46.943664 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.943638 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:46.944778 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.944755 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig" pod "78256170-0e88-45a6-84aa-f4c81d35d722" (UID: "78256170-0e88-45a6-84aa-f4c81d35d722"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:46.965382 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.965352 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 23:06:46.968134 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:46.968062 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038527 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm\") pod \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038596 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache\") pod \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038621 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp\" (UniqueName: \"kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp\") pod \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038648 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir\") pod \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038679 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir\") pod \"734b0726-6df3-4de9-b57d-1dfc752b01de\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038704 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm\") pod \"734b0726-6df3-4de9-b57d-1dfc752b01de\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " Oct 29 23:06:47.038773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038740 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs\" (UniqueName: \"kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs\") pod \"734b0726-6df3-4de9-b57d-1dfc752b01de\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038816 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs\") pod \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038902 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home\") pod \"734b0726-6df3-4de9-b57d-1dfc752b01de\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038905 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache" pod "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" (UID: "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.038954 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache\") pod \"734b0726-6df3-4de9-b57d-1dfc752b01de\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039032 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg\" (UniqueName: \"kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg\") pod \"734b0726-6df3-4de9-b57d-1dfc752b01de\" (UID: \"734b0726-6df3-4de9-b57d-1dfc752b01de\") " Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039065 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home\") pod \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\" (UID: \"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7\") " Oct 29 23:06:47.039364 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039130 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir" pod "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" (UID: "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.039761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039468 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache" pod "734b0726-6df3-4de9-b57d-1dfc752b01de" (UID: "734b0726-6df3-4de9-b57d-1dfc752b01de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.039827 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039796 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home" pod "734b0726-6df3-4de9-b57d-1dfc752b01de" (UID: "734b0726-6df3-4de9-b57d-1dfc752b01de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.039827 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039806 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home" pod "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" (UID: "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.039939 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.039816 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir" pod "734b0726-6df3-4de9-b57d-1dfc752b01de" (UID: "734b0726-6df3-4de9-b57d-1dfc752b01de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.040193 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040169 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040198 5073 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-kubeconfig\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040215 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040228 5073 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-dns-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040240 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040253 5073 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78256170-0e88-45a6-84aa-f4c81d35d722-cacerts\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040265 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrgg\" (UniqueName: \"kubernetes.io/projected/78256170-0e88-45a6-84aa-f4c81d35d722-kube-api-access-dlrgg\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040279 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040304 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040294 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040724 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040305 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040724 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040373 5073 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78256170-0e88-45a6-84aa-f4c81d35d722-istio-csr-ca-configmap\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040724 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040422 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.040724 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.040458 5073 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78256170-0e88-45a6-84aa-f4c81d35d722-local-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.041919 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.041887 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm" pod "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" (UID: "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.042478 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.042454 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs" pod "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" (UID: "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:47.042678 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.042649 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm" pod "734b0726-6df3-4de9-b57d-1dfc752b01de" (UID: "734b0726-6df3-4de9-b57d-1dfc752b01de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:47.042766 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.042698 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs" pod "734b0726-6df3-4de9-b57d-1dfc752b01de" (UID: "734b0726-6df3-4de9-b57d-1dfc752b01de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:47.044126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.044106 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp" pod "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" (UID: "6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7"). InnerVolumeSpecName "kube-api-access-jtxlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:47.044685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.044664 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg" pod "734b0726-6df3-4de9-b57d-1dfc752b01de" (UID: "734b0726-6df3-4de9-b57d-1dfc752b01de"). InnerVolumeSpecName "kube-api-access-r4shg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:47.076514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.076484 5073 scope.go:122] "RemoveContainer" containerID="9ec61e0c5a0ae0f4b52f02b4e5c72e8e57f1230bd3a23dc855c9890bca4fc105" Oct 29 23:06:47.089075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.089049 5073 scope.go:122] "RemoveContainer" containerID="5056e022f3bc80c82830dcf9ee1c8b503add1d2843abcd89133344e3ef7029fc" Oct 29 23:06:47.099748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.099726 5073 scope.go:122] "RemoveContainer" containerID="03906e1f8c098c0777bd269557010aebe9ac266aa5cef02740cae6532556c41b" Oct 29 23:06:47.108981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.108956 5073 scope.go:122] "RemoveContainer" containerID="d8b0e975f858fedec0b44c15a31668d10aac1b1ea5542326ca5704b489e4f853" Oct 29 23:06:47.117636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.117608 5073 scope.go:122] "RemoveContainer" containerID="ca33d9c2928fdd2568f36bac571013c2cf906c8aa902cf0706ca5e065ebf4671" Oct 29 23:06:47.126282 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.126266 5073 scope.go:122] "RemoveContainer" containerID="6a72c9bb8971c8359f5ba7ef84776a49e96dd324df53b466f9f7811d6dd307b5" Oct 29 23:06:47.141738 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.141706 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.141871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.141742 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4shg\" (UniqueName: \"kubernetes.io/projected/734b0726-6df3-4de9-b57d-1dfc752b01de-kube-api-access-r4shg\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.141871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.141755 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.141871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.141767 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtxlp\" (UniqueName: \"kubernetes.io/projected/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7-kube-api-access-jtxlp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.141871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.141779 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734b0726-6df3-4de9-b57d-1dfc752b01de-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.141871 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.141793 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734b0726-6df3-4de9-b57d-1dfc752b01de-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.197719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.197688 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" event={"ID":"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7","Type":"ContainerDied","Data":"03906e1f8c098c0777bd269557010aebe9ac266aa5cef02740cae6532556c41b"} Oct 29 23:06:47.197834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.197746 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" event={"ID":"6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7","Type":"ContainerDied","Data":"1a3ad88f09bb73594d7526c5f7c556a91e89de7a05ac5346f12b94b31c8e2d7e"} Oct 29 23:06:47.199200 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199172 5073 generic.go:356] "Generic (PLEG): container finished" podID="78256170-0e88-45a6-84aa-f4c81d35d722" containerID="e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420" exitCode=0 Oct 29 23:06:47.199294 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199222 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" Oct 29 23:06:47.199294 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199249 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" event={"ID":"78256170-0e88-45a6-84aa-f4c81d35d722","Type":"ContainerDied","Data":"e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420"} Oct 29 23:06:47.199383 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199299 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw" event={"ID":"78256170-0e88-45a6-84aa-f4c81d35d722","Type":"ContainerDied","Data":"fc87623e20246e15000a7f8c89068873164beca4e54f46885ba0822fe1a5009b"} Oct 29 23:06:47.199383 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199318 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" event={"ID":"734b0726-6df3-4de9-b57d-1dfc752b01de","Type":"ContainerDied","Data":"5056e022f3bc80c82830dcf9ee1c8b503add1d2843abcd89133344e3ef7029fc"} Oct 29 23:06:47.199383 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199331 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" event={"ID":"734b0726-6df3-4de9-b57d-1dfc752b01de","Type":"ContainerDied","Data":"cbda20ef7aef547736a344b4d1ca297819bff264a2fe13fea87dd8ff08dcdccf"} Oct 29 23:06:47.199383 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199347 5073 scope.go:122] "RemoveContainer" containerID="e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420" Oct 29 23:06:47.199690 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199535 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl" Oct 29 23:06:47.199690 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.199550 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" Oct 29 23:06:47.210273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.210250 5073 scope.go:122] "RemoveContainer" containerID="e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420" Oct 29 23:06:47.210567 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:47.210545 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420\": container with ID starting with e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420 not found: ID does not exist" containerID="e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420" Oct 29 23:06:47.210625 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.210578 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420"} err="failed to get container status \"e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420\": rpc error: code = NotFound desc = could not find container \"e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420\": container with ID starting with e5328d48b747dd0471efa96d1626bf6b8cbda4fb2971d80d61ed62d642e14420 not found: ID does not exist" Oct 29 23:06:47.282175 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.281449 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj"] Oct 29 23:06:47.285622 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.285589 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj"] Oct 29 23:06:47.353920 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.353830 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl"] Oct 29 23:06:47.362138 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.362110 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/simulator-w6-92fd86-kserve-cbbc5bfc6-9nmcl"] Oct 29 23:06:47.396325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.396296 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw"] Oct 29 23:06:47.401565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.401542 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lkrpw"] Oct 29 23:06:47.611530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.611454 5073 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:47.611530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.611516 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:47.867286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:47.867200 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/premium-w6-92fd86-kserve-69cf98bf6f-hhtkj" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:06:48.838608 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:48.838567 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" path="/var/lib/kubelet/pods/6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7/volumes" Oct 29 23:06:48.839218 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:48.839197 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" path="/var/lib/kubelet/pods/734b0726-6df3-4de9-b57d-1dfc752b01de/volumes" Oct 29 23:06:48.839691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:48.839671 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" path="/var/lib/kubelet/pods/78256170-0e88-45a6-84aa-f4c81d35d722/volumes" Oct 29 23:06:50.720649 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.720625 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:06:50.879765 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879721 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.879937 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879827 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.879937 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879845 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.879937 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879864 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.879937 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879915 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.880167 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879946 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.880167 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879978 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.880167 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.879983 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.880167 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880009 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.880167 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880062 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo\") pod \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\" (UID: \"0d4c6d2a-b299-45a9-a9c9-06497966fe7f\") " Oct 29 23:06:50.880167 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880133 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.880481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880243 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.880481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880289 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.880481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880370 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:50.880652 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880484 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.880652 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880499 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.880652 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880513 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.880652 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880522 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.880652 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.880533 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.882599 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.882572 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:50.882690 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.882619 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.882690 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.882646 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "kube-api-access-mrkzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.882690 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.882671 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token" pod "0d4c6d2a-b299-45a9-a9c9-06497966fe7f" (UID: "0d4c6d2a-b299-45a9-a9c9-06497966fe7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.956685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.956648 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k"] Oct 29 23:06:50.961140 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.961117 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g"] Oct 29 23:06:50.970921 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.970897 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt"] Oct 29 23:06:50.981712 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.981683 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrkzn\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-kube-api-access-mrkzn\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.981877 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.981717 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.981877 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.981732 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.981877 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:50.981745 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4c6d2a-b299-45a9-a9c9-06497966fe7f-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:06:51.237462 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.237371 5073 generic.go:356] "Generic (PLEG): container finished" podID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerID="98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a" exitCode=0 Oct 29 23:06:51.237462 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.237415 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" event={"ID":"0d4c6d2a-b299-45a9-a9c9-06497966fe7f","Type":"ContainerDied","Data":"98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a"} Oct 29 23:06:51.237462 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.237421 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" Oct 29 23:06:51.237684 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.237468 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f" event={"ID":"0d4c6d2a-b299-45a9-a9c9-06497966fe7f","Type":"ContainerDied","Data":"38b7149fc85ede4b2d5dd6e6693dbb323d854033c504b9ddb5e1006aaa1fde57"} Oct 29 23:06:51.237684 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.237498 5073 scope.go:122] "RemoveContainer" containerID="98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a" Oct 29 23:06:51.249115 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.249094 5073 scope.go:122] "RemoveContainer" containerID="98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a" Oct 29 23:06:51.249459 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:06:51.249434 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a\": container with ID starting with 98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a not found: ID does not exist" containerID="98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a" Oct 29 23:06:51.249588 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.249464 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a"} err="failed to get container status \"98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a\": rpc error: code = NotFound desc = could not find container \"98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a\": container with ID starting with 98b0f50c4a9bea2eae20621ae0348cac4ca6b303279cfe3b72274df8bc55f82a not found: ID does not exist" Oct 29 23:06:51.298874 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.298825 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f"] Oct 29 23:06:51.302080 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:51.302055 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-kzw2f"] Oct 29 23:06:52.840637 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:52.840604 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" path="/var/lib/kubelet/pods/0d4c6d2a-b299-45a9-a9c9-06497966fe7f/volumes" Oct 29 23:06:53.001988 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:53.001956 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:06:55.726332 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:55.726290 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4"] Oct 29 23:06:55.726884 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:55.726523 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" containerID="cri-o://51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad" gracePeriod=30 Oct 29 23:06:58.268661 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:58.262534 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2"] Oct 29 23:06:58.268661 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:58.262760 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" containerID="cri-o://db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804" gracePeriod=30 Oct 29 23:06:59.068956 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:59.068918 5073 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:59.069170 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:06:59.068983 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:07:01.018074 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.018049 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:07:01.089513 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089482 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089525 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089578 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089619 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089698 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089662 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089952 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089754 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089952 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089787 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.089952 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089790 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089952 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089883 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.089952 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.089933 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs\") pod \"77dbabf1-951f-42d5-b02f-363085127b26\" (UID: \"77dbabf1-951f-42d5-b02f-363085127b26\") " Oct 29 23:07:01.091078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.090207 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:01.091078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.090429 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.091078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.090915 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.091078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.090940 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.091078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.090952 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dbabf1-951f-42d5-b02f-363085127b26-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.091559 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.091530 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.091719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.091698 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.092956 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.092931 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.093589 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.093567 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:01.094073 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.094052 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.094173 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.094068 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw" pod "77dbabf1-951f-42d5-b02f-363085127b26" (UID: "77dbabf1-951f-42d5-b02f-363085127b26"). InnerVolumeSpecName "kube-api-access-n7vjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.176089 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.176059 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc"] Oct 29 23:07:01.191707 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.191682 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.191821 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.191710 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dbabf1-951f-42d5-b02f-363085127b26-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.191821 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.191725 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.191821 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.191740 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.191821 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.191755 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dbabf1-951f-42d5-b02f-363085127b26-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.191821 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.191768 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7vjw\" (UniqueName: \"kubernetes.io/projected/77dbabf1-951f-42d5-b02f-363085127b26-kube-api-access-n7vjw\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.196734 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.196710 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l"] Oct 29 23:07:01.294443 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.294353 5073 generic.go:356] "Generic (PLEG): container finished" podID="77dbabf1-951f-42d5-b02f-363085127b26" containerID="51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad" exitCode=0 Oct 29 23:07:01.294443 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.294405 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" Oct 29 23:07:01.294443 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.294418 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" event={"ID":"77dbabf1-951f-42d5-b02f-363085127b26","Type":"ContainerDied","Data":"51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad"} Oct 29 23:07:01.294691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.294470 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4" event={"ID":"77dbabf1-951f-42d5-b02f-363085127b26","Type":"ContainerDied","Data":"09da10a33bfed7577352c7e5bc3c2f4e8874d71d4709861c29cc5df851be2078"} Oct 29 23:07:01.294691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.294493 5073 scope.go:122] "RemoveContainer" containerID="51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad" Oct 29 23:07:01.305473 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.305450 5073 scope.go:122] "RemoveContainer" containerID="51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad" Oct 29 23:07:01.305752 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:01.305730 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad\": container with ID starting with 51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad not found: ID does not exist" containerID="51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad" Oct 29 23:07:01.305822 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.305764 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad"} err="failed to get container status \"51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad\": rpc error: code = NotFound desc = could not find container \"51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad\": container with ID starting with 51aaa62265e87c74042264807881c6462b760c6aacbe77693f247b6cfe3630ad not found: ID does not exist" Oct 29 23:07:01.354052 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.354013 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4"] Oct 29 23:07:01.361448 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:01.361418 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-fwdn4"] Oct 29 23:07:02.837630 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:02.837598 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77dbabf1-951f-42d5-b02f-363085127b26" path="/var/lib/kubelet/pods/77dbabf1-951f-42d5-b02f-363085127b26/volumes" Oct 29 23:07:03.545447 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.545388 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 23:07:03.715997 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.715959 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716009 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716047 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716188 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716255 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716293 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716343 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716418 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716417 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716651 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716429 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.716651 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716473 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket\") pod \"5c5e7540-5da4-48b2-8316-20b92125aa78\" (UID: \"5c5e7540-5da4-48b2-8316-20b92125aa78\") " Oct 29 23:07:03.716651 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716585 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:03.716651 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716574 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.716991 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.716815 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.717103 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.717014 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.717197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.717173 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.717268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.717203 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.717268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.717222 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.717268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.717236 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e7540-5da4-48b2-8316-20b92125aa78-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.717268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.717248 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.718565 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.718536 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "kube-api-access-hzrd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:03.718665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.718584 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:03.719124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.719104 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:03.719124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.719113 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy" pod "5c5e7540-5da4-48b2-8316-20b92125aa78" (UID: "5c5e7540-5da4-48b2-8316-20b92125aa78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.818017 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.817956 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrd9\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-kube-api-access-hzrd9\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.818017 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.817980 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.818017 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.817990 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.818017 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:03.817998 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5e7540-5da4-48b2-8316-20b92125aa78-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:04.312589 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.312556 5073 generic.go:356] "Generic (PLEG): container finished" podID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerID="db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804" exitCode=0 Oct 29 23:07:04.313077 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.312609 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" Oct 29 23:07:04.313077 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.312625 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" event={"ID":"5c5e7540-5da4-48b2-8316-20b92125aa78","Type":"ContainerDied","Data":"db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804"} Oct 29 23:07:04.313077 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.312685 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2" event={"ID":"5c5e7540-5da4-48b2-8316-20b92125aa78","Type":"ContainerDied","Data":"a7cd4e20361f0a41b7c189dd4035bd7c002dcfb9976e6ac7612668c99e43c9b8"} Oct 29 23:07:04.313077 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.312714 5073 scope.go:122] "RemoveContainer" containerID="db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804" Oct 29 23:07:04.322922 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.322901 5073 scope.go:122] "RemoveContainer" containerID="db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804" Oct 29 23:07:04.323233 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:04.323208 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804\": container with ID starting with db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804 not found: ID does not exist" containerID="db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804" Oct 29 23:07:04.323315 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.323242 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804"} err="failed to get container status \"db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804\": rpc error: code = NotFound desc = could not find container \"db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804\": container with ID starting with db46c006ec0da55af1a7d19bc9c128bd482511f97979a273fea9e4c9e36bb804 not found: ID does not exist" Oct 29 23:07:04.380909 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.380884 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2"] Oct 29 23:07:04.393895 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.393859 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-npjv2"] Oct 29 23:07:04.837786 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:04.837756 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" path="/var/lib/kubelet/pods/5c5e7540-5da4-48b2-8316-20b92125aa78/volumes" Oct 29 23:07:05.978505 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:05.978462 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" containerID="cri-o://c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0" gracePeriod=45 Oct 29 23:07:05.981675 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:05.981635 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" containerID="cri-o://791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4" gracePeriod=45 Oct 29 23:07:05.993640 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:05.993579 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" containerID="cri-o://0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90" gracePeriod=45 Oct 29 23:07:05.998624 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:05.998583 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": EOF" Oct 29 23:07:06.144944 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.144697 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused" Oct 29 23:07:06.251283 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.251258 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 23:07:06.284670 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.284645 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 23:07:06.287981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.287957 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 23:07:06.329900 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.329865 5073 generic.go:356] "Generic (PLEG): container finished" podID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerID="791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4" exitCode=0 Oct 29 23:07:06.330088 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.329922 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" Oct 29 23:07:06.330088 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.329952 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" event={"ID":"30771aa7-e03f-4557-94e0-7b920bdc68ee","Type":"ContainerDied","Data":"791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4"} Oct 29 23:07:06.330088 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.330005 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g" event={"ID":"30771aa7-e03f-4557-94e0-7b920bdc68ee","Type":"ContainerDied","Data":"d3de7e926174029803856c7f25d86e30d28f02ce3969d1d390f64d760dbf21d2"} Oct 29 23:07:06.330088 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.330032 5073 scope.go:122] "RemoveContainer" containerID="791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4" Oct 29 23:07:06.332694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.332667 5073 generic.go:356] "Generic (PLEG): container finished" podID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerID="0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90" exitCode=0 Oct 29 23:07:06.332842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.332716 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" Oct 29 23:07:06.332842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.332782 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" event={"ID":"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79","Type":"ContainerDied","Data":"0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90"} Oct 29 23:07:06.332842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.332817 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt" event={"ID":"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79","Type":"ContainerDied","Data":"dce93024aa016d83e6656bec2558e0a12d74238722efc697a83699988e4bc811"} Oct 29 23:07:06.342791 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.342758 5073 generic.go:356] "Generic (PLEG): container finished" podID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerID="c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0" exitCode=0 Oct 29 23:07:06.342920 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.342892 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" event={"ID":"6c21edbe-94a0-43f4-80c8-49d95cc292b5","Type":"ContainerDied","Data":"c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0"} Oct 29 23:07:06.342966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.342930 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" event={"ID":"6c21edbe-94a0-43f4-80c8-49d95cc292b5","Type":"ContainerDied","Data":"e92e6450331584b6828216c928f7f9367ea0097ebbd2e8d34f36ee98902617a9"} Oct 29 23:07:06.343018 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.343003 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k" Oct 29 23:07:06.349951 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.349375 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768\" (UniqueName: \"kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768\") pod \"30771aa7-e03f-4557-94e0-7b920bdc68ee\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " Oct 29 23:07:06.349951 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.349492 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs\") pod \"30771aa7-e03f-4557-94e0-7b920bdc68ee\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " Oct 29 23:07:06.349951 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.349536 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache\") pod \"30771aa7-e03f-4557-94e0-7b920bdc68ee\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " Oct 29 23:07:06.349951 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.349636 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm\") pod \"30771aa7-e03f-4557-94e0-7b920bdc68ee\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " Oct 29 23:07:06.349951 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.349698 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home\") pod \"30771aa7-e03f-4557-94e0-7b920bdc68ee\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " Oct 29 23:07:06.349951 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.349730 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir\") pod \"30771aa7-e03f-4557-94e0-7b920bdc68ee\" (UID: \"30771aa7-e03f-4557-94e0-7b920bdc68ee\") " Oct 29 23:07:06.350676 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.350231 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir" pod "30771aa7-e03f-4557-94e0-7b920bdc68ee" (UID: "30771aa7-e03f-4557-94e0-7b920bdc68ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.350676 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.350271 5073 scope.go:122] "RemoveContainer" containerID="791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4" Oct 29 23:07:06.350676 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.350354 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache" pod "30771aa7-e03f-4557-94e0-7b920bdc68ee" (UID: "30771aa7-e03f-4557-94e0-7b920bdc68ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.350863 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.350840 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home" pod "30771aa7-e03f-4557-94e0-7b920bdc68ee" (UID: "30771aa7-e03f-4557-94e0-7b920bdc68ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.351136 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:06.351102 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4\": container with ID starting with 791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4 not found: ID does not exist" containerID="791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4" Oct 29 23:07:06.351292 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.351145 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4"} err="failed to get container status \"791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4\": rpc error: code = NotFound desc = could not find container \"791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4\": container with ID starting with 791ff6a84fa6ba0583850a606f60d5c9199d8026f880c1cbc3ca82b38ff033c4 not found: ID does not exist" Oct 29 23:07:06.351292 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.351190 5073 scope.go:122] "RemoveContainer" containerID="0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90" Oct 29 23:07:06.353317 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.353293 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm" pod "30771aa7-e03f-4557-94e0-7b920bdc68ee" (UID: "30771aa7-e03f-4557-94e0-7b920bdc68ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.356454 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.356428 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768" pod "30771aa7-e03f-4557-94e0-7b920bdc68ee" (UID: "30771aa7-e03f-4557-94e0-7b920bdc68ee"). InnerVolumeSpecName "kube-api-access-vm768". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:06.361140 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.361115 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs" pod "30771aa7-e03f-4557-94e0-7b920bdc68ee" (UID: "30771aa7-e03f-4557-94e0-7b920bdc68ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:06.363526 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.363501 5073 scope.go:122] "RemoveContainer" containerID="0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90" Oct 29 23:07:06.363872 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:06.363845 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90\": container with ID starting with 0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90 not found: ID does not exist" containerID="0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90" Oct 29 23:07:06.363963 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.363878 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90"} err="failed to get container status \"0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90\": rpc error: code = NotFound desc = could not find container \"0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90\": container with ID starting with 0be8ed1308b158bf48a29486daa85d8579d8f03c2cc8e175ba3cce4691d1ae90 not found: ID does not exist" Oct 29 23:07:06.363963 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.363898 5073 scope.go:122] "RemoveContainer" containerID="c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0" Oct 29 23:07:06.373457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.373437 5073 scope.go:122] "RemoveContainer" containerID="c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0" Oct 29 23:07:06.373775 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:06.373748 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0\": container with ID starting with c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0 not found: ID does not exist" containerID="c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0" Oct 29 23:07:06.373851 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.373774 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0"} err="failed to get container status \"c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0\": rpc error: code = NotFound desc = could not find container \"c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0\": container with ID starting with c89eda01cc39f8a5c28167641113c8ac73145e59e7d609844d249b5c02cec4a0 not found: ID does not exist" Oct 29 23:07:06.450744 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450715 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache\") pod \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " Oct 29 23:07:06.450929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450758 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir\") pod \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " Oct 29 23:07:06.450929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450785 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home\") pod \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " Oct 29 23:07:06.450929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450818 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir\") pod \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " Oct 29 23:07:06.450929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450841 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm\") pod \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " Oct 29 23:07:06.450929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450869 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm\") pod \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " Oct 29 23:07:06.450929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450926 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs\") pod \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.450985 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache\") pod \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451004 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache" pod "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" (UID: "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451028 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir" pod "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" (UID: "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451056 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt\" (UniqueName: \"kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt\") pod \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451087 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs\") pod \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\" (UID: \"9a81ab41-82ec-4eb1-852f-c8d5ce53fa79\") " Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451113 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6\" (UniqueName: \"kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6\") pod \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451125 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir" pod "6c21edbe-94a0-43f4-80c8-49d95cc292b5" (UID: "6c21edbe-94a0-43f4-80c8-49d95cc292b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451134 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home" pod "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" (UID: "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.451295 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451243 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home\") pod \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\" (UID: \"6c21edbe-94a0-43f4-80c8-49d95cc292b5\") " Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451307 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache" pod "6c21edbe-94a0-43f4-80c8-49d95cc292b5" (UID: "6c21edbe-94a0-43f4-80c8-49d95cc292b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451616 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home" pod "6c21edbe-94a0-43f4-80c8-49d95cc292b5" (UID: "6c21edbe-94a0-43f4-80c8-49d95cc292b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451637 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451658 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451672 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451685 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm768\" (UniqueName: \"kubernetes.io/projected/30771aa7-e03f-4557-94e0-7b920bdc68ee-kube-api-access-vm768\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451698 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30771aa7-e03f-4557-94e0-7b920bdc68ee-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451711 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451724 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451738 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451748 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451760 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.451800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.451771 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30771aa7-e03f-4557-94e0-7b920bdc68ee-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.453440 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.453416 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm" pod "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" (UID: "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.453590 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.453440 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs" pod "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" (UID: "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:06.453888 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.453867 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6" pod "6c21edbe-94a0-43f4-80c8-49d95cc292b5" (UID: "6c21edbe-94a0-43f4-80c8-49d95cc292b5"). InnerVolumeSpecName "kube-api-access-dgps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:06.453981 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.453884 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt" pod "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" (UID: "9a81ab41-82ec-4eb1-852f-c8d5ce53fa79"). InnerVolumeSpecName "kube-api-access-pnsdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:06.454104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.454085 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm" pod "6c21edbe-94a0-43f4-80c8-49d95cc292b5" (UID: "6c21edbe-94a0-43f4-80c8-49d95cc292b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:06.454754 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.454737 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs" pod "6c21edbe-94a0-43f4-80c8-49d95cc292b5" (UID: "6c21edbe-94a0-43f4-80c8-49d95cc292b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552306 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsdt\" (UniqueName: \"kubernetes.io/projected/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-kube-api-access-pnsdt\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552333 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552345 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgps6\" (UniqueName: \"kubernetes.io/projected/6c21edbe-94a0-43f4-80c8-49d95cc292b5-kube-api-access-dgps6\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552353 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552363 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552370 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c21edbe-94a0-43f4-80c8-49d95cc292b5-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.552386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.552377 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c21edbe-94a0-43f4-80c8-49d95cc292b5-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:06.684409 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.684380 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g"] Oct 29 23:07:06.697522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.697497 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-w4-4a0287-kserve-6db95d94bf-v7r9g"] Oct 29 23:07:06.727759 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.727732 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k"] Oct 29 23:07:06.729593 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.729571 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/distinct-2-w4-4a0287-kserve-bbfdd578b-ksm6k"] Oct 29 23:07:06.763124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.763093 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt"] Oct 29 23:07:06.769020 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.768999 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a0287/premium-w4-4a0287-kserve-686596b5f4-cf6kt"] Oct 29 23:07:06.837487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.837426 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" path="/var/lib/kubelet/pods/30771aa7-e03f-4557-94e0-7b920bdc68ee/volumes" Oct 29 23:07:06.837843 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.837830 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" path="/var/lib/kubelet/pods/6c21edbe-94a0-43f4-80c8-49d95cc292b5/volumes" Oct 29 23:07:06.838232 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:06.838219 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" path="/var/lib/kubelet/pods/9a81ab41-82ec-4eb1-852f-c8d5ce53fa79/volumes" Oct 29 23:07:09.255252 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:09.255210 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:07:12.602328 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:12.602293 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m"] Oct 29 23:07:12.602835 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:12.602541 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" podUID="4e538166-0db6-4b8b-b304-0543406f5e01" containerName="maas-api" containerID="cri-o://2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623" gracePeriod=30 Oct 29 23:07:12.871322 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:12.871297 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 23:07:13.006963 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.006931 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:07:13.016633 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.016607 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca\") pod \"4e538166-0db6-4b8b-b304-0543406f5e01\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " Oct 29 23:07:13.016877 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.016845 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs\") pod \"4e538166-0db6-4b8b-b304-0543406f5e01\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " Oct 29 23:07:13.016983 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.016910 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\") pod \"4e538166-0db6-4b8b-b304-0543406f5e01\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " Oct 29 23:07:13.016983 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.016947 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs\" (UniqueName: \"kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs\") pod \"4e538166-0db6-4b8b-b304-0543406f5e01\" (UID: \"4e538166-0db6-4b8b-b304-0543406f5e01\") " Oct 29 23:07:13.016983 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.016970 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca" pod "4e538166-0db6-4b8b-b304-0543406f5e01" (UID: "4e538166-0db6-4b8b-b304-0543406f5e01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:13.017324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.017289 5073 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e538166-0db6-4b8b-b304-0543406f5e01-openshift-service-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.019409 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.019313 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls" pod "4e538166-0db6-4b8b-b304-0543406f5e01" (UID: "4e538166-0db6-4b8b-b304-0543406f5e01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:13.019533 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.019500 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs" pod "4e538166-0db6-4b8b-b304-0543406f5e01" (UID: "4e538166-0db6-4b8b-b304-0543406f5e01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:13.019622 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.019601 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs" pod "4e538166-0db6-4b8b-b304-0543406f5e01" (UID: "4e538166-0db6-4b8b-b304-0543406f5e01"). InnerVolumeSpecName "kube-api-access-c7lcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:13.118208 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.118115 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-metrics-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.118208 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.118142 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e538166-0db6-4b8b-b304-0543406f5e01-maas-api-tls\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.118208 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.118199 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7lcs\" (UniqueName: \"kubernetes.io/projected/4e538166-0db6-4b8b-b304-0543406f5e01-kube-api-access-c7lcs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.381428 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.381341 5073 generic.go:356] "Generic (PLEG): container finished" podID="4e538166-0db6-4b8b-b304-0543406f5e01" containerID="2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623" exitCode=0 Oct 29 23:07:13.381428 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.381388 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" Oct 29 23:07:13.381630 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.381423 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" event={"ID":"4e538166-0db6-4b8b-b304-0543406f5e01","Type":"ContainerDied","Data":"2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623"} Oct 29 23:07:13.381630 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.381481 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m" event={"ID":"4e538166-0db6-4b8b-b304-0543406f5e01","Type":"ContainerDied","Data":"0e3e777a2f5f69efcbb024c45e081ff09d51d53cb2c951e594c44b0b63d51126"} Oct 29 23:07:13.381630 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.381505 5073 scope.go:122] "RemoveContainer" containerID="2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623" Oct 29 23:07:13.391199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.391182 5073 scope.go:122] "RemoveContainer" containerID="2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623" Oct 29 23:07:13.391497 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:13.391477 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623\": container with ID starting with 2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623 not found: ID does not exist" containerID="2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623" Oct 29 23:07:13.391549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.391505 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623"} err="failed to get container status \"2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623\": rpc error: code = NotFound desc = could not find container \"2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623\": container with ID starting with 2206f5d0ed21302d4ff519685bd09277c46436f1739f09f85613becbb83ae623 not found: ID does not exist" Oct 29 23:07:13.435240 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.435210 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m"] Oct 29 23:07:13.448617 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:13.448591 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a0287-86d496664d-gn46m"] Oct 29 23:07:14.837285 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:14.837251 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e538166-0db6-4b8b-b304-0543406f5e01" path="/var/lib/kubelet/pods/4e538166-0db6-4b8b-b304-0543406f5e01/volumes" Oct 29 23:07:16.197819 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.197776 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" containerID="cri-o://1a670d7b0355ae01d4697635036f3aabf628cb265963fd2e33618c1eaa88d7b0" gracePeriod=45 Oct 29 23:07:16.223583 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.223547 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" containerID="cri-o://fde199883627442e34b2807f229e49b3e5beb22562cca96723ccf8880929707f" gracePeriod=45 Oct 29 23:07:16.404444 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.404415 5073 generic.go:356] "Generic (PLEG): container finished" podID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerID="1a670d7b0355ae01d4697635036f3aabf628cb265963fd2e33618c1eaa88d7b0" exitCode=0 Oct 29 23:07:16.404581 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.404486 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" event={"ID":"aaec2da7-e3b0-4000-adc5-bba4afca80a1","Type":"ContainerDied","Data":"1a670d7b0355ae01d4697635036f3aabf628cb265963fd2e33618c1eaa88d7b0"} Oct 29 23:07:16.406252 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.406221 5073 generic.go:356] "Generic (PLEG): container finished" podID="fde5beee-062c-4a4a-9993-998e693ffe17" containerID="fde199883627442e34b2807f229e49b3e5beb22562cca96723ccf8880929707f" exitCode=0 Oct 29 23:07:16.406350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.406282 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" event={"ID":"fde5beee-062c-4a4a-9993-998e693ffe17","Type":"ContainerDied","Data":"fde199883627442e34b2807f229e49b3e5beb22562cca96723ccf8880929707f"} Oct 29 23:07:16.484500 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.484477 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 23:07:16.509137 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.509111 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 23:07:16.548398 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548370 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache\") pod \"fde5beee-062c-4a4a-9993-998e693ffe17\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " Oct 29 23:07:16.548566 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548405 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm\") pod \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " Oct 29 23:07:16.548566 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548436 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir\") pod \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " Oct 29 23:07:16.548566 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548513 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home\") pod \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " Oct 29 23:07:16.548748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548692 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache" pod "fde5beee-062c-4a4a-9993-998e693ffe17" (UID: "fde5beee-062c-4a4a-9993-998e693ffe17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.548748 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548706 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir" pod "aaec2da7-e3b0-4000-adc5-bba4afca80a1" (UID: "aaec2da7-e3b0-4000-adc5-bba4afca80a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.548861 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548782 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home" pod "aaec2da7-e3b0-4000-adc5-bba4afca80a1" (UID: "aaec2da7-e3b0-4000-adc5-bba4afca80a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.548861 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548549 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs\" (UniqueName: \"kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs\") pod \"fde5beee-062c-4a4a-9993-998e693ffe17\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " Oct 29 23:07:16.548984 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548872 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs\") pod \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " Oct 29 23:07:16.548984 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548897 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm\") pod \"fde5beee-062c-4a4a-9993-998e693ffe17\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " Oct 29 23:07:16.548984 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.548918 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home\") pod \"fde5beee-062c-4a4a-9993-998e693ffe17\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " Oct 29 23:07:16.549545 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.549496 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home" pod "fde5beee-062c-4a4a-9993-998e693ffe17" (UID: "fde5beee-062c-4a4a-9993-998e693ffe17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.549839 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm\" (UniqueName: \"kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm\") pod \"fde5beee-062c-4a4a-9993-998e693ffe17\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.549896 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2\" (UniqueName: \"kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2\") pod \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.549955 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir\") pod \"fde5beee-062c-4a4a-9993-998e693ffe17\" (UID: \"fde5beee-062c-4a4a-9993-998e693ffe17\") " Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.549988 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache\") pod \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\" (UID: \"aaec2da7-e3b0-4000-adc5-bba4afca80a1\") " Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.550436 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.550459 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.550473 5073 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-home\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.550488 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.550692 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache" pod "aaec2da7-e3b0-4000-adc5-bba4afca80a1" (UID: "aaec2da7-e3b0-4000-adc5-bba4afca80a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.553128 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.551210 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir" pod "fde5beee-062c-4a4a-9993-998e693ffe17" (UID: "fde5beee-062c-4a4a-9993-998e693ffe17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.553678 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.553600 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs" pod "fde5beee-062c-4a4a-9993-998e693ffe17" (UID: "fde5beee-062c-4a4a-9993-998e693ffe17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:16.553719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.553684 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm" pod "aaec2da7-e3b0-4000-adc5-bba4afca80a1" (UID: "aaec2da7-e3b0-4000-adc5-bba4afca80a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.553880 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.553853 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs" pod "aaec2da7-e3b0-4000-adc5-bba4afca80a1" (UID: "aaec2da7-e3b0-4000-adc5-bba4afca80a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:16.554008 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.553989 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm" pod "fde5beee-062c-4a4a-9993-998e693ffe17" (UID: "fde5beee-062c-4a4a-9993-998e693ffe17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:16.554255 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.554234 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2" pod "aaec2da7-e3b0-4000-adc5-bba4afca80a1" (UID: "aaec2da7-e3b0-4000-adc5-bba4afca80a1"). InnerVolumeSpecName "kube-api-access-8r6c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:16.555553 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.555531 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm" pod "fde5beee-062c-4a4a-9993-998e693ffe17" (UID: "fde5beee-062c-4a4a-9993-998e693ffe17"). InnerVolumeSpecName "kube-api-access-xw7bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:16.651348 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651314 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651348 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651344 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde5beee-062c-4a4a-9993-998e693ffe17-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651358 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaec2da7-e3b0-4000-adc5-bba4afca80a1-tls-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651369 5073 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-dshm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651382 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7bm\" (UniqueName: \"kubernetes.io/projected/fde5beee-062c-4a4a-9993-998e693ffe17-kube-api-access-xw7bm\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651394 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r6c2\" (UniqueName: \"kubernetes.io/projected/aaec2da7-e3b0-4000-adc5-bba4afca80a1-kube-api-access-8r6c2\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651405 5073 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde5beee-062c-4a4a-9993-998e693ffe17-tmp-dir\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:16.651572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:16.651417 5073 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaec2da7-e3b0-4000-adc5-bba4afca80a1-model-cache\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:17.413560 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.413525 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" Oct 29 23:07:17.414119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.413525 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc" event={"ID":"aaec2da7-e3b0-4000-adc5-bba4afca80a1","Type":"ContainerDied","Data":"2e9a612abff4c71f37e0c985fdde82fe176568a56824ed650397671388b8c424"} Oct 29 23:07:17.414119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.413656 5073 scope.go:122] "RemoveContainer" containerID="1a670d7b0355ae01d4697635036f3aabf628cb265963fd2e33618c1eaa88d7b0" Oct 29 23:07:17.415204 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.415183 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" Oct 29 23:07:17.415314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.415203 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l" event={"ID":"fde5beee-062c-4a4a-9993-998e693ffe17","Type":"ContainerDied","Data":"8719ba17a0860d79d144ef92d538c329ad9ea49724d607d4ce2ccd1f9f96af5c"} Oct 29 23:07:17.423670 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.423653 5073 scope.go:122] "RemoveContainer" containerID="fde199883627442e34b2807f229e49b3e5beb22562cca96723ccf8880929707f" Oct 29 23:07:17.469614 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.469559 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l"] Oct 29 23:07:17.473788 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.473763 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/premium-w3-95672c-kserve-558d9c9f65-9927l"] Oct 29 23:07:17.509368 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.509342 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc"] Oct 29 23:07:17.512589 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:17.512563 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-95672c/distinct-2-w3-95672c-kserve-78b776cbcf-6mbmc"] Oct 29 23:07:18.836408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:18.836376 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" path="/var/lib/kubelet/pods/aaec2da7-e3b0-4000-adc5-bba4afca80a1/volumes" Oct 29 23:07:18.836800 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:18.836769 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" path="/var/lib/kubelet/pods/fde5beee-062c-4a4a-9993-998e693ffe17/volumes" Oct 29 23:07:20.045533 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:20.045499 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr"] Oct 29 23:07:20.046399 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:20.046363 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" containerID="cri-o://42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f" gracePeriod=30 Oct 29 23:07:25.233519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.233492 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 23:07:25.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331035 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331072 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331095 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331130 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331165 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331194 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331214 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331540 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331354 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.331540 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331367 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331540 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331475 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket\") pod \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\" (UID: \"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a\") " Oct 29 23:07:25.331540 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331489 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.331741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331546 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.331741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331675 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:25.332083 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331902 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.332083 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331929 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.332083 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.331978 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.332083 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.332009 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.332407 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.332056 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.333607 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.333578 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:25.333713 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.333617 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:25.333713 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.333638 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "kube-api-access-lzz4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:25.333713 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.333643 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo" pod "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" (UID: "3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:25.433119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.433088 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.433119 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.433115 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzz4j\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-kube-api-access-lzz4j\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.433306 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.433127 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.433306 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.433136 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.433306 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.433143 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.461498 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.461468 5073 generic.go:356] "Generic (PLEG): container finished" podID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerID="42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f" exitCode=0 Oct 29 23:07:25.461609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.461524 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" Oct 29 23:07:25.461609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.461553 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" event={"ID":"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a","Type":"ContainerDied","Data":"42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f"} Oct 29 23:07:25.461609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.461607 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr" event={"ID":"3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a","Type":"ContainerDied","Data":"8f624f2b0758d5d96e812354dc055d4f6267a95dd04b4098b67df2ec21fb469d"} Oct 29 23:07:25.461736 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.461630 5073 scope.go:122] "RemoveContainer" containerID="42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f" Oct 29 23:07:25.473657 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.473540 5073 scope.go:122] "RemoveContainer" containerID="42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f" Oct 29 23:07:25.474113 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:25.474090 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f\": container with ID starting with 42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f not found: ID does not exist" containerID="42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f" Oct 29 23:07:25.474228 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.474127 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f"} err="failed to get container status \"42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f\": rpc error: code = NotFound desc = could not find container \"42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f\": container with ID starting with 42876a74c1f8dea1527d4dc81cb40a41897a99a4645921fb2d92352895951e5f not found: ID does not exist" Oct 29 23:07:25.515520 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.515493 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr"] Oct 29 23:07:25.519433 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:25.519414 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-t8xfr"] Oct 29 23:07:26.837159 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:26.837124 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" path="/var/lib/kubelet/pods/3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a/volumes" Oct 29 23:07:27.200311 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:27.200223 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:07:28.168799 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:28.168767 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl"] Oct 29 23:07:28.169587 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:28.169554 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" containerID="cri-o://127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797" gracePeriod=30 Oct 29 23:07:29.720710 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:29.720638 5073 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:07:29.720710 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:29.720705 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:07:33.435351 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.435326 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 23:07:33.506273 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.506238 5073 generic.go:356] "Generic (PLEG): container finished" podID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerID="127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797" exitCode=0 Oct 29 23:07:33.506453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.506295 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" Oct 29 23:07:33.506453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.506337 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" event={"ID":"d2f24af6-6096-4896-8ddd-e0083eee2132","Type":"ContainerDied","Data":"127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797"} Oct 29 23:07:33.506453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.506395 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl" event={"ID":"d2f24af6-6096-4896-8ddd-e0083eee2132","Type":"ContainerDied","Data":"d985e15ba91b9bc28738f34c5ac46a56fb0b6c9e83c0eaffad3e4335935b9608"} Oct 29 23:07:33.506453 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.506425 5073 scope.go:122] "RemoveContainer" containerID="127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797" Oct 29 23:07:33.513882 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.513853 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514033 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.513902 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514033 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514015 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514164 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514048 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514164 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514102 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514164 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514135 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514285 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514196 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514285 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514228 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514368 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514295 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs\") pod \"d2f24af6-6096-4896-8ddd-e0083eee2132\" (UID: \"d2f24af6-6096-4896-8ddd-e0083eee2132\") " Oct 29 23:07:33.514828 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514804 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.515008 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.514937 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.515389 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.515363 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.517248 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.516572 5073 scope.go:122] "RemoveContainer" containerID="127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797" Oct 29 23:07:33.517248 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.517064 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.517248 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:33.517090 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797\": container with ID starting with 127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797 not found: ID does not exist" containerID="127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797" Oct 29 23:07:33.517248 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.517128 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797"} err="failed to get container status \"127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797\": rpc error: code = NotFound desc = could not find container \"127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797\": container with ID starting with 127e2184efeb4ebbc0e81260203e23d141b1b3b6fbeab63f4c1727f7eed47797 not found: ID does not exist" Oct 29 23:07:33.517614 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.517543 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:33.518143 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.518118 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:33.518287 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.518195 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "kube-api-access-nxt24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:33.518456 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.518431 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:33.518943 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.518918 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy" pod "d2f24af6-6096-4896-8ddd-e0083eee2132" (UID: "d2f24af6-6096-4896-8ddd-e0083eee2132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:33.615405 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615361 5073 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-data\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615405 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615393 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxt24\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-kube-api-access-nxt24\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615405 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615403 5073 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-token\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615405 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615413 5073 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-podinfo\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615765 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615421 5073 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-credential-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615765 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615434 5073 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f24af6-6096-4896-8ddd-e0083eee2132-istiod-ca-cert\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615765 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615445 5073 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-certs\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615765 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615453 5073 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-istio-envoy\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.615765 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.615460 5073 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f24af6-6096-4896-8ddd-e0083eee2132-workload-socket\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:07:33.854161 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.854118 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl"] Oct 29 23:07:33.867918 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:33.867894 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-m9bfl"] Oct 29 23:07:34.838268 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:34.838235 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" path="/var/lib/kubelet/pods/d2f24af6-6096-4896-8ddd-e0083eee2132/volumes" Oct 29 23:07:39.255688 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:07:39.255648 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:07:49.330276 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330241 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564"] Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330611 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330622 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330634 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330639 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330655 5073 cpu_manager.go:414] "containerMap: removing container" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" containerName="discovery" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330660 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" containerName="discovery" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330667 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330672 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330677 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330682 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330689 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" Oct 29 23:07:49.330691 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330695 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330702 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5a4278c6-0bad-4065-b53a-5b886753d055" containerName="revoke-keys" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330707 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4278c6-0bad-4065-b53a-5b886753d055" containerName="revoke-keys" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330717 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4e538166-0db6-4b8b-b304-0543406f5e01" containerName="maas-api" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330722 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e538166-0db6-4b8b-b304-0543406f5e01" containerName="maas-api" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330730 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330735 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330741 5073 cpu_manager.go:414] "containerMap: removing container" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330745 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330751 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330755 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330764 5073 cpu_manager.go:414] "containerMap: removing container" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330768 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330777 5073 cpu_manager.go:414] "containerMap: removing container" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330781 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330788 5073 cpu_manager.go:414] "containerMap: removing container" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330792 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330799 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330805 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330810 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330828 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330883 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e44bd4e-871d-48b2-9c8d-ea4a692f9ff7" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330890 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4278c6-0bad-4065-b53a-5b886753d055" containerName="revoke-keys" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330897 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e538166-0db6-4b8b-b304-0543406f5e01" containerName="maas-api" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330903 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4c6d2a-b299-45a9-a9c9-06497966fe7f" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330909 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b0726-6df3-4de9-b57d-1dfc752b01de" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330914 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7448d27-6f4d-4020-9525-7f82d08a3806" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330920 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaec2da7-e3b0-4000-adc5-bba4afca80a1" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330925 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="77dbabf1-951f-42d5-b02f-363085127b26" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330930 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="30771aa7-e03f-4557-94e0-7b920bdc68ee" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330935 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde5beee-062c-4a4a-9993-998e693ffe17" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330940 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a81ab41-82ec-4eb1-852f-c8d5ce53fa79" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330946 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5e7540-5da4-48b2-8316-20b92125aa78" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330951 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbe2a41-e6cc-4f1f-bfe0-d77a9ba7958a" containerName="istio-proxy" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330957 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="78256170-0e88-45a6-84aa-f4c81d35d722" containerName="discovery" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330961 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c21edbe-94a0-43f4-80c8-49d95cc292b5" containerName="main" Oct 29 23:07:49.331096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.330965 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f24af6-6096-4896-8ddd-e0083eee2132" containerName="istio-proxy" Oct 29 23:07:49.335570 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.335554 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.360863 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.360822 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564"] Oct 29 23:07:49.462281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462244 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462281 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462280 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462541 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462308 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wfb\" (UniqueName: \"kubernetes.io/projected/2ed97708-4f01-4e75-a35e-9e901a923009-kube-api-access-x8wfb\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462541 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462468 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed97708-4f01-4e75-a35e-9e901a923009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462541 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462526 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462693 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462570 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed97708-4f01-4e75-a35e-9e901a923009-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462693 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462586 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462693 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462643 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed97708-4f01-4e75-a35e-9e901a923009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.462808 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.462699 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564145 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564108 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed97708-4f01-4e75-a35e-9e901a923009-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564145 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564146 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564208 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed97708-4f01-4e75-a35e-9e901a923009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564240 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564296 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564318 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564344 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wfb\" (UniqueName: \"kubernetes.io/projected/2ed97708-4f01-4e75-a35e-9e901a923009-kube-api-access-x8wfb\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564386 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed97708-4f01-4e75-a35e-9e901a923009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564763 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564412 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564763 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564701 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564763 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564718 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564910 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564785 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.564910 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.564902 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.565084 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.565067 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed97708-4f01-4e75-a35e-9e901a923009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.566805 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.566778 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed97708-4f01-4e75-a35e-9e901a923009-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.566958 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.566943 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed97708-4f01-4e75-a35e-9e901a923009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.585256 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.585194 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed97708-4f01-4e75-a35e-9e901a923009-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.585377 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.585298 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wfb\" (UniqueName: \"kubernetes.io/projected/2ed97708-4f01-4e75-a35e-9e901a923009-kube-api-access-x8wfb\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-bb564\" (UID: \"2ed97708-4f01-4e75-a35e-9e901a923009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.665357 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.665330 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:49.982048 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.982025 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564"] Oct 29 23:07:49.984192 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:07:49.984162 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed97708_4f01_4e75_a35e_9e901a923009.slice/crio-378e3bf8544ab3c07937525560de37222663c6ac46492879d6cf3bdeacc9d1d9 WatchSource:0}: Error finding container 378e3bf8544ab3c07937525560de37222663c6ac46492879d6cf3bdeacc9d1d9: Status 404 returned error can't find the container with id 378e3bf8544ab3c07937525560de37222663c6ac46492879d6cf3bdeacc9d1d9 Oct 29 23:07:49.986324 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.986288 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:07:49.986434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.986366 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:07:49.986434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:49.986411 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:07:50.594357 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:50.594321 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" event={"ID":"2ed97708-4f01-4e75-a35e-9e901a923009","Type":"ContainerStarted","Data":"43d4d479266df5ccbcd855bbc855f96a6b4d94e303387d8b932bca3fae8322b4"} Oct 29 23:07:50.594753 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:50.594377 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" event={"ID":"2ed97708-4f01-4e75-a35e-9e901a923009","Type":"ContainerStarted","Data":"378e3bf8544ab3c07937525560de37222663c6ac46492879d6cf3bdeacc9d1d9"} Oct 29 23:07:50.639583 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:50.639529 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podStartSLOduration=1.639512522 podStartE2EDuration="1.639512522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:50.637484285 +0000 UTC m=+3185.952878412" watchObservedRunningTime="2026-10-29 23:07:50.639512522 +0000 UTC m=+3185.954906602" Oct 29 23:07:50.666199 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:50.666140 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:07:51.667355 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:51.667323 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:07:51.667761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:51.667370 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:07:52.666665 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:52.666625 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:07:52.666844 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:52.666697 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:07:53.665961 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:53.665928 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:07:53.666426 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:53.665978 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:07:54.666347 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:54.666269 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:07:54.666347 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:54.666333 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:07:55.666063 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:55.666024 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:07:55.666298 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:55.666087 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:07:56.666480 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:56.666445 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:07:56.666905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:56.666498 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:07:58.667109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:58.667059 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:07:58.667728 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:58.667116 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:59.708336 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:59.708286 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:07:59.709078 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:07:59.708363 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:00.179837 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:00.179750 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:08:00.180229 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:00.180198 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:08:00.208474 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:00.208449 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" Oct 29 23:08:04.376576 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.376545 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk"] Oct 29 23:08:04.380215 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.380194 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.404750 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.404720 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk"] Oct 29 23:08:04.426278 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426247 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426445 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426294 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426445 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426321 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426445 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426378 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426467 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426529 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426574 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426628 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7lw\" (UniqueName: \"kubernetes.io/projected/818c7229-c7ee-447d-b2fe-c9421d2e0fff-kube-api-access-xf7lw\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.426869 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.426677 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527297 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527331 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527487 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527347 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527476 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527519 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527594 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527680 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527644 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527900 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527696 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7lw\" (UniqueName: \"kubernetes.io/projected/818c7229-c7ee-447d-b2fe-c9421d2e0fff-kube-api-access-xf7lw\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527900 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527745 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527900 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527746 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.527900 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527809 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.528085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.527973 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.528085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.528000 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.528085 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.528051 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.530127 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.530104 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.530369 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.530351 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.548773 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.548744 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818c7229-c7ee-447d-b2fe-c9421d2e0fff-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.549177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.549129 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7lw\" (UniqueName: \"kubernetes.io/projected/818c7229-c7ee-447d-b2fe-c9421d2e0fff-kube-api-access-xf7lw\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk\" (UID: \"818c7229-c7ee-447d-b2fe-c9421d2e0fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:04.694794 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:04.694727 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:05.021320 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.021293 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk"] Oct 29 23:08:05.022927 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:08:05.022901 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818c7229_c7ee_447d_b2fe_c9421d2e0fff.slice/crio-3eff6a4082548633d21515d76104d29dd10b7abbf15807be25ddbee93434e97a WatchSource:0}: Error finding container 3eff6a4082548633d21515d76104d29dd10b7abbf15807be25ddbee93434e97a: Status 404 returned error can't find the container with id 3eff6a4082548633d21515d76104d29dd10b7abbf15807be25ddbee93434e97a Oct 29 23:08:05.024996 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.024964 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:08:05.025082 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.025027 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:08:05.025082 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.025056 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 23:08:05.692130 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.692097 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" event={"ID":"818c7229-c7ee-447d-b2fe-c9421d2e0fff","Type":"ContainerStarted","Data":"dc0945e9ea384056d9217d15bf98f3bf6df6c3615b7f1956d4c604233c193113"} Oct 29 23:08:05.692564 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.692165 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" event={"ID":"818c7229-c7ee-447d-b2fe-c9421d2e0fff","Type":"ContainerStarted","Data":"3eff6a4082548633d21515d76104d29dd10b7abbf15807be25ddbee93434e97a"} Oct 29 23:08:05.695455 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.695434 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:05.770761 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:05.770708 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podStartSLOduration=1.770694809 podStartE2EDuration="1.770694809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:05.756116863 +0000 UTC m=+3201.071510942" watchObservedRunningTime="2026-10-29 23:08:05.770694809 +0000 UTC m=+3201.086088889" Oct 29 23:08:06.696754 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:06.696713 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.697264 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:06.696772 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:08:07.696081 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:07.696044 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:08:07.696286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:07.696097 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:08:08.695396 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:08.695357 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:08:08.695803 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:08.695418 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:08:09.259303 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:08:09.259257 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:08:09.695922 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:09.695845 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:08:09.695922 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:09.695898 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:08:10.695309 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:10.695267 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:08:10.695509 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:10.695336 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:08:11.695914 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:11.695880 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:08:11.696464 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:11.695947 5073 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:08:13.254904 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:13.254852 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:13.255522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:13.255257 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:13.256262 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:13.256228 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" Oct 29 23:08:39.255126 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:08:39.255076 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:08:40.934165 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:40.934119 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:08:58.114023 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:08:58.113988 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:09:02.837260 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:02.837228 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:09:09.254760 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:09:09.254721 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:09:18.940088 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:18.940043 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:09:34.495313 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:34.495225 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:09:37.222297 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:37.222257 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:09:37.222915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:37.222880 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp" podUID="9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:39.253364 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:09:39.253325 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:09:45.933685 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:45.933653 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:09:45.936934 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:45.936912 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:09:45.939030 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:45.939007 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:09:45.941971 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:45.941953 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:09:57.930237 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:09:57.930202 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:10:08.315063 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:10:08.315016 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:10:09.253681 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:10:09.253643 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:10:12.513883 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:10:12.513850 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:10:29.702016 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:10:29.701980 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:10:34.886605 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:10:34.886558 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:10:39.254234 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:10:39.254195 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:10:51.295127 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:10:51.295089 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:10:55.497090 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:10:55.497006 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:11:09.257853 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:11:09.257806 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:11:13.041905 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:13.041867 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:11:30.220377 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:30.220337 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:11:39.255978 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:11:39.255936 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:11:40.925391 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:40.925358 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:11:42.028982 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:42.028942 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn"] Oct 29 23:11:42.043408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:42.043375 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249ad63e-nbbkn"] Oct 29 23:11:42.874183 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:42.872515 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4278c6-0bad-4065-b53a-5b886753d055" path="/var/lib/kubelet/pods/5a4278c6-0bad-4065-b53a-5b886753d055/volumes" Oct 29 23:11:46.217004 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:11:46.216969 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:12:09.255350 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:12:09.255313 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:12:32.456481 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:12:32.456377 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:12:39.255045 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:12:39.255009 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:12:42.945673 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:12:42.945638 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:12:47.278520 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:12:47.278486 5073 scope.go:122] "RemoveContainer" containerID="a96cbe9c89bb6ad9600a26ad895c26d5bd03568d605bc2269a8bfd5cad7c8878" Oct 29 23:12:57.018590 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:12:57.018555 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:13:03.027839 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.027802 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z"] Oct 29 23:13:03.032866 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.032848 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.050138 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.050109 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z"] Oct 29 23:13:03.061257 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.061221 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:13:03.061388 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.061266 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:13:03.061388 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.061287 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:13:03.062006 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.061959 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:13:03.164664 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.164638 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnzl\" (UniqueName: \"kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.164774 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.164750 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.164822 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.164783 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.265135 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.265114 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.265270 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.265183 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnzl\" (UniqueName: \"kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.265270 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.265238 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.265812 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.265793 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.267672 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.267647 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.286386 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.286330 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnzl\" (UniqueName: \"kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl\") pod \"maas-api-revoke-sub-e7e7044d-g422z\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.345099 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.345077 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:03.642294 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:03.642264 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z"] Oct 29 23:13:03.644506 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:13:03.644477 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ce0489_52c2_4a3b_8572_d69fd219015d.slice/crio-e4fed29f2e71f33358ba1291f027c62f0ce430be2d09efe2e3ba7574e1ecfe11 WatchSource:0}: Error finding container e4fed29f2e71f33358ba1291f027c62f0ce430be2d09efe2e3ba7574e1ecfe11: Status 404 returned error can't find the container with id e4fed29f2e71f33358ba1291f027c62f0ce430be2d09efe2e3ba7574e1ecfe11 Oct 29 23:13:04.222143 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:04.222110 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" event={"ID":"16ce0489-52c2-4a3b-8572-d69fd219015d","Type":"ContainerStarted","Data":"4f5ce09ea33926dac56dfd07d9d63345f8a59f4d5e588c3966f6959f49758294"} Oct 29 23:13:04.222531 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:04.222193 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" event={"ID":"16ce0489-52c2-4a3b-8572-d69fd219015d","Type":"ContainerStarted","Data":"e4fed29f2e71f33358ba1291f027c62f0ce430be2d09efe2e3ba7574e1ecfe11"} Oct 29 23:13:04.263855 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:04.263816 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" podStartSLOduration=1.263804716 podStartE2EDuration="1.263804716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:04.262958512 +0000 UTC m=+3499.578352592" watchObservedRunningTime="2026-10-29 23:13:04.263804716 +0000 UTC m=+3499.579198795" Oct 29 23:13:05.228100 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:05.228061 5073 generic.go:356] "Generic (PLEG): container finished" podID="16ce0489-52c2-4a3b-8572-d69fd219015d" containerID="4f5ce09ea33926dac56dfd07d9d63345f8a59f4d5e588c3966f6959f49758294" exitCode=0 Oct 29 23:13:05.228548 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:05.228161 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" event={"ID":"16ce0489-52c2-4a3b-8572-d69fd219015d","Type":"ContainerDied","Data":"4f5ce09ea33926dac56dfd07d9d63345f8a59f4d5e588c3966f6959f49758294"} Oct 29 23:13:06.386781 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.386755 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:06.494915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.494838 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca\") pod \"16ce0489-52c2-4a3b-8572-d69fd219015d\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " Oct 29 23:13:06.494915 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.494890 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl\" (UniqueName: \"kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl\") pod \"16ce0489-52c2-4a3b-8572-d69fd219015d\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " Oct 29 23:13:06.495198 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.494966 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca\") pod \"16ce0489-52c2-4a3b-8572-d69fd219015d\" (UID: \"16ce0489-52c2-4a3b-8572-d69fd219015d\") " Oct 29 23:13:06.495320 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.495295 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca" pod "16ce0489-52c2-4a3b-8572-d69fd219015d" (UID: "16ce0489-52c2-4a3b-8572-d69fd219015d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:06.497299 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.497277 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl" pod "16ce0489-52c2-4a3b-8572-d69fd219015d" (UID: "16ce0489-52c2-4a3b-8572-d69fd219015d"). InnerVolumeSpecName "kube-api-access-tlnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:06.497385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.497297 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca" pod "16ce0489-52c2-4a3b-8572-d69fd219015d" (UID: "16ce0489-52c2-4a3b-8572-d69fd219015d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:06.596707 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.596682 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:13:06.596707 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.596704 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ce0489-52c2-4a3b-8572-d69fd219015d-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:13:06.596848 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:06.596715 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlnzl\" (UniqueName: \"kubernetes.io/projected/16ce0489-52c2-4a3b-8572-d69fd219015d-kube-api-access-tlnzl\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:13:07.239697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:07.239664 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" Oct 29 23:13:07.239697 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:07.239677 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z" event={"ID":"16ce0489-52c2-4a3b-8572-d69fd219015d","Type":"ContainerDied","Data":"e4fed29f2e71f33358ba1291f027c62f0ce430be2d09efe2e3ba7574e1ecfe11"} Oct 29 23:13:07.239884 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:07.239723 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4fed29f2e71f33358ba1291f027c62f0ce430be2d09efe2e3ba7574e1ecfe11" Oct 29 23:13:09.253730 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:13:09.253691 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:13:14.278238 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.278201 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq"] Oct 29 23:13:14.278641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.278601 5073 cpu_manager.go:414] "containerMap: removing container" podUID="16ce0489-52c2-4a3b-8572-d69fd219015d" containerName="revoke-keys" Oct 29 23:13:14.278641 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.278612 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce0489-52c2-4a3b-8572-d69fd219015d" containerName="revoke-keys" Oct 29 23:13:14.278721 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.278698 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce0489-52c2-4a3b-8572-d69fd219015d" containerName="revoke-keys" Oct 29 23:13:14.282109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.282090 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.308356 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.308332 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:13:14.308356 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.308341 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:13:14.308511 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.308376 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:13:14.308511 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.308413 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:13:14.323957 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.323881 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq"] Oct 29 23:13:14.364444 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.364418 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.364549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.364472 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62skn\" (UniqueName: \"kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.364549 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.364497 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.465649 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.465627 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.465770 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.465672 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62skn\" (UniqueName: \"kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.465770 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.465698 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.466340 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.466320 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:13:14.466454 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.466435 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.468313 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.468294 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.487051 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.487033 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62skn\" (UniqueName: \"kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.594703 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.594651 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:14.892395 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:14.892367 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq"] Oct 29 23:13:14.894446 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:13:14.894395 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43057046_b8bd_4f86_b48f_20be6d08ced4.slice/crio-4467428e360f8be26469e73909eb10d4fc47a1c023586470d784987c2164b56a WatchSource:0}: Error finding container 4467428e360f8be26469e73909eb10d4fc47a1c023586470d784987c2164b56a: Status 404 returned error can't find the container with id 4467428e360f8be26469e73909eb10d4fc47a1c023586470d784987c2164b56a Oct 29 23:13:15.282883 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:15.282856 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" event={"ID":"43057046-b8bd-4f86-b48f-20be6d08ced4","Type":"ContainerStarted","Data":"c8f5636346e68f60c9ba5ad786d87e084fa1646ccc7ec734558467989cb6d7c5"} Oct 29 23:13:15.283242 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:15.282914 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" event={"ID":"43057046-b8bd-4f86-b48f-20be6d08ced4","Type":"ContainerStarted","Data":"4467428e360f8be26469e73909eb10d4fc47a1c023586470d784987c2164b56a"} Oct 29 23:13:15.324343 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:15.324301 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" podStartSLOduration=1.324289385 podStartE2EDuration="1.324289385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:15.323634363 +0000 UTC m=+3510.639028443" watchObservedRunningTime="2026-10-29 23:13:15.324289385 +0000 UTC m=+3510.639683464" Oct 29 23:13:16.289894 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:16.289860 5073 generic.go:356] "Generic (PLEG): container finished" podID="43057046-b8bd-4f86-b48f-20be6d08ced4" containerID="c8f5636346e68f60c9ba5ad786d87e084fa1646ccc7ec734558467989cb6d7c5" exitCode=0 Oct 29 23:13:16.290332 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:16.289956 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" event={"ID":"43057046-b8bd-4f86-b48f-20be6d08ced4","Type":"ContainerDied","Data":"c8f5636346e68f60c9ba5ad786d87e084fa1646ccc7ec734558467989cb6d7c5"} Oct 29 23:13:17.436959 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.436936 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:17.595883 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.595810 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca\") pod \"43057046-b8bd-4f86-b48f-20be6d08ced4\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " Oct 29 23:13:17.595883 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.595876 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn\" (UniqueName: \"kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn\") pod \"43057046-b8bd-4f86-b48f-20be6d08ced4\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " Oct 29 23:13:17.596083 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.595937 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca\") pod \"43057046-b8bd-4f86-b48f-20be6d08ced4\" (UID: \"43057046-b8bd-4f86-b48f-20be6d08ced4\") " Oct 29 23:13:17.596262 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.596240 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca" pod "43057046-b8bd-4f86-b48f-20be6d08ced4" (UID: "43057046-b8bd-4f86-b48f-20be6d08ced4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:17.598223 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.598198 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca" pod "43057046-b8bd-4f86-b48f-20be6d08ced4" (UID: "43057046-b8bd-4f86-b48f-20be6d08ced4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:17.598351 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.598234 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn" pod "43057046-b8bd-4f86-b48f-20be6d08ced4" (UID: "43057046-b8bd-4f86-b48f-20be6d08ced4"). InnerVolumeSpecName "kube-api-access-62skn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:17.696872 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.696845 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:13:17.696872 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.696870 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62skn\" (UniqueName: \"kubernetes.io/projected/43057046-b8bd-4f86-b48f-20be6d08ced4-kube-api-access-62skn\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:13:17.697014 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:17.696879 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43057046-b8bd-4f86-b48f-20be6d08ced4-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:13:18.301532 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:18.301508 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" Oct 29 23:13:18.301704 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:18.301540 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq" event={"ID":"43057046-b8bd-4f86-b48f-20be6d08ced4","Type":"ContainerDied","Data":"4467428e360f8be26469e73909eb10d4fc47a1c023586470d784987c2164b56a"} Oct 29 23:13:18.301704 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:18.301605 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4467428e360f8be26469e73909eb10d4fc47a1c023586470d784987c2164b56a" Oct 29 23:13:26.189242 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:26.189208 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq"] Oct 29 23:13:26.202114 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:26.202084 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e3223-fwwfq"] Oct 29 23:13:26.836234 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:26.836200 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43057046-b8bd-4f86-b48f-20be6d08ced4" path="/var/lib/kubelet/pods/43057046-b8bd-4f86-b48f-20be6d08ced4/volumes" Oct 29 23:13:27.936497 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:13:27.936457 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:13:39.253925 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:13:39.253887 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:14:09.253245 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:14:09.253208 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:14:39.254448 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:14:39.254410 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:14:45.965712 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:14:45.965683 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:14:45.969801 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:14:45.969765 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:14:45.971125 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:14:45.971106 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:14:45.974911 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:14:45.974890 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:15:09.254741 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:15:09.254691 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:15:14.459959 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:15:14.459926 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:15:30.099197 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:15:30.099108 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:15:39.254389 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:15:39.254346 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:15:56.950184 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:15:56.950134 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:16:09.254535 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:16:09.254497 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:16:12.312448 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:12.312411 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:16:26.370053 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.370017 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4"] Oct 29 23:16:26.370465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.370447 5073 cpu_manager.go:414] "containerMap: removing container" podUID="43057046-b8bd-4f86-b48f-20be6d08ced4" containerName="revoke-keys" Oct 29 23:16:26.370465 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.370459 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43057046-b8bd-4f86-b48f-20be6d08ced4" containerName="revoke-keys" Oct 29 23:16:26.370546 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.370540 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="43057046-b8bd-4f86-b48f-20be6d08ced4" containerName="revoke-keys" Oct 29 23:16:26.373694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.373677 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.389772 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.389747 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:16:26.402701 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.402672 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:16:26.402856 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.402835 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:16:26.402901 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.402835 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:16:26.404684 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.404664 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4"] Oct 29 23:16:26.427509 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.427480 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rlb\" (UniqueName: \"kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.427636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.427528 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.427707 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.427688 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.501498 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.501467 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:16:26.528547 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.528521 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rlb\" (UniqueName: \"kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.528696 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.528582 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.528764 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.528714 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.529369 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.529348 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.531075 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.531057 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.548852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.548829 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rlb\" (UniqueName: \"kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb\") pod \"maas-api-revoke-sub-f1f2a58b-gvjs4\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.686696 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.686604 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:26.996929 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:26.996904 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4"] Oct 29 23:16:26.999545 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:16:26.999512 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53af0bb2_6b5c_4641_ac6a_063af89f1898.slice/crio-b712338d237be47b85a376fe6efcfe3df5382875fdda514dfc48a71a9f252c96 WatchSource:0}: Error finding container b712338d237be47b85a376fe6efcfe3df5382875fdda514dfc48a71a9f252c96: Status 404 returned error can't find the container with id b712338d237be47b85a376fe6efcfe3df5382875fdda514dfc48a71a9f252c96 Oct 29 23:16:27.237863 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:27.237825 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" event={"ID":"53af0bb2-6b5c-4641-ac6a-063af89f1898","Type":"ContainerStarted","Data":"b712338d237be47b85a376fe6efcfe3df5382875fdda514dfc48a71a9f252c96"} Oct 29 23:16:28.243625 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:28.243587 5073 generic.go:356] "Generic (PLEG): container finished" podID="53af0bb2-6b5c-4641-ac6a-063af89f1898" containerID="0473b19ff18dea6b6264e2c68bb2011c23f11dd9b42c7903a54ea3a895a94a33" exitCode=0 Oct 29 23:16:28.244019 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:28.243633 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" event={"ID":"53af0bb2-6b5c-4641-ac6a-063af89f1898","Type":"ContainerDied","Data":"0473b19ff18dea6b6264e2c68bb2011c23f11dd9b42c7903a54ea3a895a94a33"} Oct 29 23:16:29.388390 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.388366 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:29.456719 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.456687 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca\") pod \"53af0bb2-6b5c-4641-ac6a-063af89f1898\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " Oct 29 23:16:29.456883 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.456743 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb\" (UniqueName: \"kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb\") pod \"53af0bb2-6b5c-4641-ac6a-063af89f1898\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " Oct 29 23:16:29.456883 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.456849 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca\") pod \"53af0bb2-6b5c-4641-ac6a-063af89f1898\" (UID: \"53af0bb2-6b5c-4641-ac6a-063af89f1898\") " Oct 29 23:16:29.457106 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.457082 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca" pod "53af0bb2-6b5c-4641-ac6a-063af89f1898" (UID: "53af0bb2-6b5c-4641-ac6a-063af89f1898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:29.459178 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.459119 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca" pod "53af0bb2-6b5c-4641-ac6a-063af89f1898" (UID: "53af0bb2-6b5c-4641-ac6a-063af89f1898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:29.459290 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.459264 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb" pod "53af0bb2-6b5c-4641-ac6a-063af89f1898" (UID: "53af0bb2-6b5c-4641-ac6a-063af89f1898"). InnerVolumeSpecName "kube-api-access-t7rlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:29.557674 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.557623 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.557674 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.557643 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rlb\" (UniqueName: \"kubernetes.io/projected/53af0bb2-6b5c-4641-ac6a-063af89f1898-kube-api-access-t7rlb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.557674 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:29.557653 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af0bb2-6b5c-4641-ac6a-063af89f1898-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:16:30.254509 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:30.254482 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" Oct 29 23:16:30.254692 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:30.254474 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4" event={"ID":"53af0bb2-6b5c-4641-ac6a-063af89f1898","Type":"ContainerDied","Data":"b712338d237be47b85a376fe6efcfe3df5382875fdda514dfc48a71a9f252c96"} Oct 29 23:16:30.254692 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:30.254600 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b712338d237be47b85a376fe6efcfe3df5382875fdda514dfc48a71a9f252c96" Oct 29 23:16:37.420498 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.420469 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29"] Oct 29 23:16:37.420944 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.420851 5073 cpu_manager.go:414] "containerMap: removing container" podUID="53af0bb2-6b5c-4641-ac6a-063af89f1898" containerName="revoke-keys" Oct 29 23:16:37.420944 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.420864 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53af0bb2-6b5c-4641-ac6a-063af89f1898" containerName="revoke-keys" Oct 29 23:16:37.421052 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.420953 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="53af0bb2-6b5c-4641-ac6a-063af89f1898" containerName="revoke-keys" Oct 29 23:16:37.424174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.424128 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.439852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.439825 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:16:37.439852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.439840 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:16:37.440021 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.439923 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:16:37.453177 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.453125 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:16:37.456184 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.456127 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29"] Oct 29 23:16:37.526457 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.526427 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.526600 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.526471 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb26\" (UniqueName: \"kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.526600 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.526493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.627842 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.627808 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.627996 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.627863 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb26\" (UniqueName: \"kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.627996 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.627885 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.628550 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.628526 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.630326 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.630306 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.648272 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.648252 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb26\" (UniqueName: \"kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.737247 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.737219 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:37.827609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:37.827580 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:16:38.037388 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:38.037361 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29"] Oct 29 23:16:38.043081 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:16:38.043012 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f31b78a_1149_4ffb_9c6e_48cc60f91efb.slice/crio-2296b5f53727a7d75258f22698053e7562ae2ad85594abf530f8af0a19be1ea1 WatchSource:0}: Error finding container 2296b5f53727a7d75258f22698053e7562ae2ad85594abf530f8af0a19be1ea1: Status 404 returned error can't find the container with id 2296b5f53727a7d75258f22698053e7562ae2ad85594abf530f8af0a19be1ea1 Oct 29 23:16:38.291952 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:38.291881 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" event={"ID":"8f31b78a-1149-4ffb-9c6e-48cc60f91efb","Type":"ContainerStarted","Data":"2296b5f53727a7d75258f22698053e7562ae2ad85594abf530f8af0a19be1ea1"} Oct 29 23:16:39.254404 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:16:39.254369 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:16:39.299526 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:39.299491 5073 generic.go:356] "Generic (PLEG): container finished" podID="8f31b78a-1149-4ffb-9c6e-48cc60f91efb" containerID="776c171eff3e6aadd4e1fb7676458682fea42e08c3cb04b690e162284b92a7b3" exitCode=0 Oct 29 23:16:39.299672 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:39.299529 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" event={"ID":"8f31b78a-1149-4ffb-9c6e-48cc60f91efb","Type":"ContainerDied","Data":"776c171eff3e6aadd4e1fb7676458682fea42e08c3cb04b690e162284b92a7b3"} Oct 29 23:16:40.442124 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.442097 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:40.553351 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.553325 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca\") pod \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " Oct 29 23:16:40.553514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.553390 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca\") pod \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " Oct 29 23:16:40.553514 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.553429 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26\" (UniqueName: \"kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26\") pod \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\" (UID: \"8f31b78a-1149-4ffb-9c6e-48cc60f91efb\") " Oct 29 23:16:40.553742 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.553718 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca" pod "8f31b78a-1149-4ffb-9c6e-48cc60f91efb" (UID: "8f31b78a-1149-4ffb-9c6e-48cc60f91efb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:40.555609 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.555547 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca" pod "8f31b78a-1149-4ffb-9c6e-48cc60f91efb" (UID: "8f31b78a-1149-4ffb-9c6e-48cc60f91efb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:40.555717 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.555672 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26" pod "8f31b78a-1149-4ffb-9c6e-48cc60f91efb" (UID: "8f31b78a-1149-4ffb-9c6e-48cc60f91efb"). InnerVolumeSpecName "kube-api-access-cfb26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:40.654965 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.654939 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:16:40.654965 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.654963 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfb26\" (UniqueName: \"kubernetes.io/projected/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-kube-api-access-cfb26\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:16:40.655096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:40.654973 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31b78a-1149-4ffb-9c6e-48cc60f91efb-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:16:41.310942 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:41.310911 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" Oct 29 23:16:41.311098 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:41.310929 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29" event={"ID":"8f31b78a-1149-4ffb-9c6e-48cc60f91efb","Type":"ContainerDied","Data":"2296b5f53727a7d75258f22698053e7562ae2ad85594abf530f8af0a19be1ea1"} Oct 29 23:16:41.311098 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:41.310993 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2296b5f53727a7d75258f22698053e7562ae2ad85594abf530f8af0a19be1ea1" Oct 29 23:16:51.486319 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:51.486283 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29"] Oct 29 23:16:51.497096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:51.497067 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2250ca-vlq29"] Oct 29 23:16:52.837234 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:52.837193 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f31b78a-1149-4ffb-9c6e-48cc60f91efb" path="/var/lib/kubelet/pods/8f31b78a-1149-4ffb-9c6e-48cc60f91efb/volumes" Oct 29 23:16:53.247469 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:16:53.247434 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:17:07.217056 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:07.216962 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:17:09.295350 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:17:09.295304 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:17:30.029708 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:30.029673 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:17:39.255360 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:17:39.255317 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:17:43.598833 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:43.598800 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:17:46.234307 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:46.234244 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:46.234967 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:46.234915 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:56.411579 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:56.411537 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:17:59.300329 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:59.300290 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:59.300751 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:17:59.300362 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk" podUID="818c7229-c7ee-447d-b2fe-c9421d2e0fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:18:03.558772 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:03.558730 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:18:07.027648 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:07.027609 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z"] Oct 29 23:18:07.040644 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:07.040608 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e7044d-g422z"] Oct 29 23:18:08.836104 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:08.836070 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ce0489-52c2-4a3b-8572-d69fd219015d" path="/var/lib/kubelet/pods/16ce0489-52c2-4a3b-8572-d69fd219015d/volumes" Oct 29 23:18:09.254288 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:18:09.254255 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:18:11.404840 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:11.404803 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:18:15.726385 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:15.726348 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:18:33.995504 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:33.995424 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:18:39.253453 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:18:39.253410 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:18:43.900138 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:43.900108 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:18:46.229174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:46.228246 5073 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-bb564 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:18:46.229174 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:46.228319 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-bb564" podUID="2ed97708-4f01-4e75-a35e-9e901a923009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:18:52.196383 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:52.196350 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:18:59.172022 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.171989 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d"] Oct 29 23:18:59.172444 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.172429 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8f31b78a-1149-4ffb-9c6e-48cc60f91efb" containerName="revoke-keys" Oct 29 23:18:59.172494 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.172445 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f31b78a-1149-4ffb-9c6e-48cc60f91efb" containerName="revoke-keys" Oct 29 23:18:59.172530 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.172518 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f31b78a-1149-4ffb-9c6e-48cc60f91efb" containerName="revoke-keys" Oct 29 23:18:59.175676 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.175659 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.205397 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.205372 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:18:59.205522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.205400 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:18:59.205522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.205440 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:18:59.205522 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.205467 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:18:59.207214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.207194 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d"] Oct 29 23:18:59.216702 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.216674 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.216803 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.216714 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.216858 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.216830 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cckp\" (UniqueName: \"kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.317243 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.317218 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cckp\" (UniqueName: \"kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.317350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.317255 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.317350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.317283 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.317924 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.317902 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.319747 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.319715 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.337907 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.337880 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cckp\" (UniqueName: \"kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp\") pod \"maas-api-revoke-sub-f03629db-frz4d\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.488054 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.488034 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:18:59.785377 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:18:59.785353 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d"] Oct 29 23:18:59.787654 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:18:59.787622 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b2502b_dd53_49ee_9b60_d4adc8c0be54.slice/crio-219a7e88dea155d295c12b7281a8abd3605fbb4fd60feadaece13434a0daee9e WatchSource:0}: Error finding container 219a7e88dea155d295c12b7281a8abd3605fbb4fd60feadaece13434a0daee9e: Status 404 returned error can't find the container with id 219a7e88dea155d295c12b7281a8abd3605fbb4fd60feadaece13434a0daee9e Oct 29 23:19:00.115055 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:00.115014 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" event={"ID":"63b2502b-dd53-49ee-9b60-d4adc8c0be54","Type":"ContainerStarted","Data":"3e7b9ac3479242c5639c26882f128774956adeaa6d4cd3d9454390a9b159ee79"} Oct 29 23:19:00.115214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:00.115076 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" event={"ID":"63b2502b-dd53-49ee-9b60-d4adc8c0be54","Type":"ContainerStarted","Data":"219a7e88dea155d295c12b7281a8abd3605fbb4fd60feadaece13434a0daee9e"} Oct 29 23:19:00.155601 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:00.155557 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" podStartSLOduration=1.155546132 podStartE2EDuration="1.155546132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:00.154885093 +0000 UTC m=+3855.470279172" watchObservedRunningTime="2026-10-29 23:19:00.155546132 +0000 UTC m=+3855.470940253" Oct 29 23:19:01.120516 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:01.120483 5073 generic.go:356] "Generic (PLEG): container finished" podID="63b2502b-dd53-49ee-9b60-d4adc8c0be54" containerID="3e7b9ac3479242c5639c26882f128774956adeaa6d4cd3d9454390a9b159ee79" exitCode=0 Oct 29 23:19:01.120890 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:01.120564 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" event={"ID":"63b2502b-dd53-49ee-9b60-d4adc8c0be54","Type":"ContainerDied","Data":"3e7b9ac3479242c5639c26882f128774956adeaa6d4cd3d9454390a9b159ee79"} Oct 29 23:19:02.262091 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.262065 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:19:02.342029 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.342003 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp\" (UniqueName: \"kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp\") pod \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " Oct 29 23:19:02.342203 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.342113 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca\") pod \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " Oct 29 23:19:02.342270 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.342245 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca\") pod \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\" (UID: \"63b2502b-dd53-49ee-9b60-d4adc8c0be54\") " Oct 29 23:19:02.342572 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.342549 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca" pod "63b2502b-dd53-49ee-9b60-d4adc8c0be54" (UID: "63b2502b-dd53-49ee-9b60-d4adc8c0be54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:02.344237 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.344210 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca" pod "63b2502b-dd53-49ee-9b60-d4adc8c0be54" (UID: "63b2502b-dd53-49ee-9b60-d4adc8c0be54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:19:02.344350 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.344321 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp" pod "63b2502b-dd53-49ee-9b60-d4adc8c0be54" (UID: "63b2502b-dd53-49ee-9b60-d4adc8c0be54"). InnerVolumeSpecName "kube-api-access-7cckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:02.443906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.443856 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cckp\" (UniqueName: \"kubernetes.io/projected/63b2502b-dd53-49ee-9b60-d4adc8c0be54-kube-api-access-7cckp\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:02.443906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.443877 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:02.443906 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:02.443888 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b2502b-dd53-49ee-9b60-d4adc8c0be54-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:03.131252 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:03.131197 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" Oct 29 23:19:03.131380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:03.131247 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-frz4d" event={"ID":"63b2502b-dd53-49ee-9b60-d4adc8c0be54","Type":"ContainerDied","Data":"219a7e88dea155d295c12b7281a8abd3605fbb4fd60feadaece13434a0daee9e"} Oct 29 23:19:03.131380 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:03.131293 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="219a7e88dea155d295c12b7281a8abd3605fbb4fd60feadaece13434a0daee9e" Oct 29 23:19:05.203371 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:05.203342 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:19:09.300455 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:19:09.300420 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:19:10.822953 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.822916 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w"] Oct 29 23:19:10.823622 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.823599 5073 cpu_manager.go:414] "containerMap: removing container" podUID="63b2502b-dd53-49ee-9b60-d4adc8c0be54" containerName="revoke-keys" Oct 29 23:19:10.823667 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.823627 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b2502b-dd53-49ee-9b60-d4adc8c0be54" containerName="revoke-keys" Oct 29 23:19:10.823757 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.823745 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b2502b-dd53-49ee-9b60-d4adc8c0be54" containerName="revoke-keys" Oct 29 23:19:10.827636 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.827618 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:10.842330 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.842306 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w"] Oct 29 23:19:10.853468 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.853440 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:19:10.853600 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.853490 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:19:10.853600 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.853531 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:19:10.856129 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.856109 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:19:10.912225 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.912197 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:10.912397 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.912277 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjznb\" (UniqueName: \"kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:10.912520 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:10.912499 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.014050 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.014010 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.014236 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.014130 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjznb\" (UniqueName: \"kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.014236 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.014190 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.014725 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.014706 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.016669 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.016647 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.035358 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.035332 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjznb\" (UniqueName: \"kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb\") pod \"maas-api-revoke-sub-0165d7d2-n267w\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.142447 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.142395 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:11.444311 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:11.444286 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w"] Oct 29 23:19:11.446538 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:19:11.446511 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38002ea_cbbd_43d5_b3c0_ee3485492729.slice/crio-db2c26834d6ce41767acf12086a6fdd7e31363fb6034af66265d96c59083b3db WatchSource:0}: Error finding container db2c26834d6ce41767acf12086a6fdd7e31363fb6034af66265d96c59083b3db: Status 404 returned error can't find the container with id db2c26834d6ce41767acf12086a6fdd7e31363fb6034af66265d96c59083b3db Oct 29 23:19:12.194109 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:12.194077 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" event={"ID":"a38002ea-cbbd-43d5-b3c0-ee3485492729","Type":"ContainerStarted","Data":"320ea157578ce3bfa8ac4332fe63b545c5aac31439cf3c9a1b12f4ed4d785b7f"} Oct 29 23:19:12.194519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:12.194136 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" event={"ID":"a38002ea-cbbd-43d5-b3c0-ee3485492729","Type":"ContainerStarted","Data":"db2c26834d6ce41767acf12086a6fdd7e31363fb6034af66265d96c59083b3db"} Oct 29 23:19:12.235494 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:12.235439 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" podStartSLOduration=2.235425953 podStartE2EDuration="2.235425953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:12.234107841 +0000 UTC m=+3867.549501938" watchObservedRunningTime="2026-10-29 23:19:12.235425953 +0000 UTC m=+3867.550820031" Oct 29 23:19:13.200382 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:13.200349 5073 generic.go:356] "Generic (PLEG): container finished" podID="a38002ea-cbbd-43d5-b3c0-ee3485492729" containerID="320ea157578ce3bfa8ac4332fe63b545c5aac31439cf3c9a1b12f4ed4d785b7f" exitCode=0 Oct 29 23:19:13.200802 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:13.200430 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" event={"ID":"a38002ea-cbbd-43d5-b3c0-ee3485492729","Type":"ContainerDied","Data":"320ea157578ce3bfa8ac4332fe63b545c5aac31439cf3c9a1b12f4ed4d785b7f"} Oct 29 23:19:14.330427 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.330403 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:14.441545 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.441522 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca\") pod \"a38002ea-cbbd-43d5-b3c0-ee3485492729\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " Oct 29 23:19:14.441660 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.441558 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb\" (UniqueName: \"kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb\") pod \"a38002ea-cbbd-43d5-b3c0-ee3485492729\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " Oct 29 23:19:14.441660 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.441607 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca\") pod \"a38002ea-cbbd-43d5-b3c0-ee3485492729\" (UID: \"a38002ea-cbbd-43d5-b3c0-ee3485492729\") " Oct 29 23:19:14.441873 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.441851 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca" pod "a38002ea-cbbd-43d5-b3c0-ee3485492729" (UID: "a38002ea-cbbd-43d5-b3c0-ee3485492729"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:14.443792 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.443748 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb" pod "a38002ea-cbbd-43d5-b3c0-ee3485492729" (UID: "a38002ea-cbbd-43d5-b3c0-ee3485492729"). InnerVolumeSpecName "kube-api-access-kjznb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:14.443885 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.443825 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca" pod "a38002ea-cbbd-43d5-b3c0-ee3485492729" (UID: "a38002ea-cbbd-43d5-b3c0-ee3485492729"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:19:14.542414 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.542387 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:14.542414 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.542412 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjznb\" (UniqueName: \"kubernetes.io/projected/a38002ea-cbbd-43d5-b3c0-ee3485492729-kube-api-access-kjznb\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:14.542557 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:14.542426 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38002ea-cbbd-43d5-b3c0-ee3485492729-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:15.211096 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:15.211051 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" event={"ID":"a38002ea-cbbd-43d5-b3c0-ee3485492729","Type":"ContainerDied","Data":"db2c26834d6ce41767acf12086a6fdd7e31363fb6034af66265d96c59083b3db"} Oct 29 23:19:15.211252 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:15.211113 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2c26834d6ce41767acf12086a6fdd7e31363fb6034af66265d96c59083b3db" Oct 29 23:19:15.211252 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:15.211071 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n267w" Oct 29 23:19:21.813117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.813087 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q"] Oct 29 23:19:21.813519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.813503 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a38002ea-cbbd-43d5-b3c0-ee3485492729" containerName="revoke-keys" Oct 29 23:19:21.813519 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.813517 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38002ea-cbbd-43d5-b3c0-ee3485492729" containerName="revoke-keys" Oct 29 23:19:21.813621 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.813611 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38002ea-cbbd-43d5-b3c0-ee3485492729" containerName="revoke-keys" Oct 29 23:19:21.816737 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.816715 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:21.848294 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.848268 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:19:21.848434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.848270 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:19:21.848434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.848361 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:19:21.848434 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.848392 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:19:21.850423 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.850397 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q"] Oct 29 23:19:21.903325 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.903297 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grnwl\" (UniqueName: \"kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:21.903477 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.903355 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:21.903477 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:21.903374 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.003867 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.003841 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.003989 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.003870 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.003989 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.003927 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grnwl\" (UniqueName: \"kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.004480 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.004461 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.006303 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.006284 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.025179 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.025143 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grnwl\" (UniqueName: \"kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl\") pod \"maas-api-revoke-sub-6b19338f-grx9q\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.129836 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.129787 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:22.428264 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:22.428232 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q"] Oct 29 23:19:22.429456 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:19:22.429428 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622c1f43_c6fd_42b0_8895_cd02f1e7af77.slice/crio-49f399a668c7d7fc9607bcc39cea256ada15d8ca90231a88a821fc73b83d15f9 WatchSource:0}: Error finding container 49f399a668c7d7fc9607bcc39cea256ada15d8ca90231a88a821fc73b83d15f9: Status 404 returned error can't find the container with id 49f399a668c7d7fc9607bcc39cea256ada15d8ca90231a88a821fc73b83d15f9 Oct 29 23:19:23.251256 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:23.251224 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" event={"ID":"622c1f43-c6fd-42b0-8895-cd02f1e7af77","Type":"ContainerStarted","Data":"b81c376c0a8983726063bb843c47c930fa7351d37be5e83ec54988e06cf924d0"} Oct 29 23:19:23.251659 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:23.251291 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" event={"ID":"622c1f43-c6fd-42b0-8895-cd02f1e7af77","Type":"ContainerStarted","Data":"49f399a668c7d7fc9607bcc39cea256ada15d8ca90231a88a821fc73b83d15f9"} Oct 29 23:19:24.258126 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.258095 5073 generic.go:356] "Generic (PLEG): container finished" podID="622c1f43-c6fd-42b0-8895-cd02f1e7af77" containerID="b81c376c0a8983726063bb843c47c930fa7351d37be5e83ec54988e06cf924d0" exitCode=0 Oct 29 23:19:24.258634 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.258187 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" event={"ID":"622c1f43-c6fd-42b0-8895-cd02f1e7af77","Type":"ContainerDied","Data":"b81c376c0a8983726063bb843c47c930fa7351d37be5e83ec54988e06cf924d0"} Oct 29 23:19:24.395818 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.395796 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:24.527314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.527252 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca\") pod \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " Oct 29 23:19:24.527314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.527292 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl\" (UniqueName: \"kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl\") pod \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " Oct 29 23:19:24.527314 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.527311 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca\") pod \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\" (UID: \"622c1f43-c6fd-42b0-8895-cd02f1e7af77\") " Oct 29 23:19:24.527657 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.527633 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca" pod "622c1f43-c6fd-42b0-8895-cd02f1e7af77" (UID: "622c1f43-c6fd-42b0-8895-cd02f1e7af77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:24.529508 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.529480 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl" pod "622c1f43-c6fd-42b0-8895-cd02f1e7af77" (UID: "622c1f43-c6fd-42b0-8895-cd02f1e7af77"). InnerVolumeSpecName "kube-api-access-grnwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:24.529613 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.529502 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca" pod "622c1f43-c6fd-42b0-8895-cd02f1e7af77" (UID: "622c1f43-c6fd-42b0-8895-cd02f1e7af77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:19:24.628833 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.628808 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-openshift-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:24.628833 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.628831 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grnwl\" (UniqueName: \"kubernetes.io/projected/622c1f43-c6fd-42b0-8895-cd02f1e7af77-kube-api-access-grnwl\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:24.628966 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:24.628840 5073 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622c1f43-c6fd-42b0-8895-cd02f1e7af77-maas-api-xks-ca\") on node \"ip-10-0-134-94.ec2.internal\" DevicePath \"\"" Oct 29 23:19:25.264107 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:25.264078 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" Oct 29 23:19:25.264502 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:25.264110 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-grx9q" event={"ID":"622c1f43-c6fd-42b0-8895-cd02f1e7af77","Type":"ContainerDied","Data":"49f399a668c7d7fc9607bcc39cea256ada15d8ca90231a88a821fc73b83d15f9"} Oct 29 23:19:25.264502 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:25.264211 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f399a668c7d7fc9607bcc39cea256ada15d8ca90231a88a821fc73b83d15f9" Oct 29 23:19:33.113004 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:33.112971 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pw8wr"] Oct 29 23:19:39.254210 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:19:39.254171 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:19:44.229129 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:44.229088 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zlw9_30070c77-9435-4c71-b525-5fbc18e94ed7/manager/0.log" Oct 29 23:19:44.440494 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:44.440465 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-lw9wk_3e4bda76-7201-4d14-b501-79f936a6fb1b/manager/0.log" Oct 29 23:19:44.672431 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:44.672378 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ldwdt_551636ca-bee8-45ca-96b3-0e57666a5e6c/manager/0.log" Oct 29 23:19:45.596870 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:45.596840 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dg9sk_a0e90837-ba5f-4c9d-9f0c-839609878ed9/server/0.log" Oct 29 23:19:45.797670 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:45.797647 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-2fxfz_5ce9a9c1-c283-4138-b25b-f4bff5aee593/manager/0.log" Oct 29 23:19:45.998937 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:45.998915 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:19:46.003602 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:46.003579 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:19:46.003828 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:46.003812 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:19:46.013812 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:46.013785 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hx452_0762bfee-d0ac-4a88-bee5-212f3b230a2d/ovn-acl-logging/0.log" Oct 29 23:19:46.532844 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:46.532822 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-nrbsf_e839cdff-da2e-41f3-9755-f15b5a4dfc95/copy-manifests/0.log" Oct 29 23:19:46.568466 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:46.568437 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-nrbsf_e839cdff-da2e-41f3-9755-f15b5a4dfc95/manager/0.log" Oct 29 23:19:47.344342 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:47.344312 5073 scope.go:122] "RemoveContainer" containerID="4f5ce09ea33926dac56dfd07d9d63345f8a59f4d5e588c3966f6959f49758294" Oct 29 23:19:47.353289 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:47.353267 5073 scope.go:122] "RemoveContainer" containerID="c8f5636346e68f60c9ba5ad786d87e084fa1646ccc7ec734558467989cb6d7c5" Oct 29 23:19:48.210048 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:48.210021 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-n267w_a38002ea-cbbd-43d5-b3c0-ee3485492729/revoke-keys/0.log" Oct 29 23:19:48.767658 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:48.767637 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-grx9q_622c1f43-c6fd-42b0-8895-cd02f1e7af77/revoke-keys/0.log" Oct 29 23:19:49.530087 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:49.530062 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-frz4d_63b2502b-dd53-49ee-9b60-d4adc8c0be54/revoke-keys/0.log" Oct 29 23:19:49.718736 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:49.718704 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f1f2a58b-gvjs4_53af0bb2-6b5c-4641-ac6a-063af89f1898/revoke-keys/0.log" Oct 29 23:19:52.661600 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:52.661572 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pw8wr_68657adb-f9ad-417d-a37d-2252e0fecb41/limitador/0.log" Oct 29 23:19:53.661240 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:53.661214 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-pzncp_0e30adc0-b639-4735-94ab-c80c7d97b200/istio-proxy/0.log" Oct 29 23:19:54.456032 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:54.455969 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-bb564_2ed97708-4f01-4e75-a35e-9e901a923009/istio-proxy/0.log" Oct 29 23:19:55.479286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:55.479261 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk_818c7229-c7ee-447d-b2fe-c9421d2e0fff/istio-proxy/0.log" Oct 29 23:19:55.677701 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:55.677676 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp_9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a/istio-proxy/0.log" Oct 29 23:19:56.860424 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:19:56.860398 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b64c8c7d5-7zxkb_7a66a3fc-f006-419b-9442-ef4d8da8bd5e/router/0.log" Oct 29 23:20:08.990785 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:08.990745 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zlw9_30070c77-9435-4c71-b525-5fbc18e94ed7/manager/0.log" Oct 29 23:20:09.199490 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:09.199462 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-lw9wk_3e4bda76-7201-4d14-b501-79f936a6fb1b/manager/0.log" Oct 29 23:20:09.254517 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:20:09.254449 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:20:09.417886 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:09.417860 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ldwdt_551636ca-bee8-45ca-96b3-0e57666a5e6c/manager/0.log" Oct 29 23:20:09.882681 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:09.882657 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dg9sk_a0e90837-ba5f-4c9d-9f0c-839609878ed9/server/0.log" Oct 29 23:20:10.079449 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:10.079425 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-2fxfz_5ce9a9c1-c283-4138-b25b-f4bff5aee593/manager/0.log" Oct 29 23:20:10.800541 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:10.800515 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-nrbsf_e839cdff-da2e-41f3-9755-f15b5a4dfc95/copy-manifests/0.log" Oct 29 23:20:10.835046 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:10.835018 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-nrbsf_e839cdff-da2e-41f3-9755-f15b5a4dfc95/manager/0.log" Oct 29 23:20:12.219755 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:12.219729 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-n267w_a38002ea-cbbd-43d5-b3c0-ee3485492729/revoke-keys/0.log" Oct 29 23:20:12.787513 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:12.787482 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-grx9q_622c1f43-c6fd-42b0-8895-cd02f1e7af77/revoke-keys/0.log" Oct 29 23:20:13.555360 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:13.555331 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-frz4d_63b2502b-dd53-49ee-9b60-d4adc8c0be54/revoke-keys/0.log" Oct 29 23:20:13.744682 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:13.744660 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f1f2a58b-gvjs4_53af0bb2-6b5c-4641-ac6a-063af89f1898/revoke-keys/0.log" Oct 29 23:20:16.121720 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:16.121694 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pw8wr_68657adb-f9ad-417d-a37d-2252e0fecb41/limitador/0.log" Oct 29 23:20:16.861730 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:16.861704 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-pzncp_0e30adc0-b639-4735-94ab-c80c7d97b200/istio-proxy/0.log" Oct 29 23:20:17.628142 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:17.628115 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-bb564_2ed97708-4f01-4e75-a35e-9e901a923009/istio-proxy/0.log" Oct 29 23:20:18.629634 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:18.629595 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk_818c7229-c7ee-447d-b2fe-c9421d2e0fff/istio-proxy/0.log" Oct 29 23:20:18.832741 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:18.832708 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp_9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a/istio-proxy/0.log" Oct 29 23:20:20.002846 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:20:20.002807 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b64c8c7d5-7zxkb_7a66a3fc-f006-419b-9442-ef4d8da8bd5e/router/0.log" Oct 29 23:20:39.254336 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:20:39.254297 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:21:09.254374 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:21:09.254339 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:21:30.025661 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:30.025566 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4"] Oct 29 23:21:30.038408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:30.038375 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f2a58b-gvjs4"] Oct 29 23:21:30.836655 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:30.836616 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53af0bb2-6b5c-4641-ac6a-063af89f1898" path="/var/lib/kubelet/pods/53af0bb2-6b5c-4641-ac6a-063af89f1898/volumes" Oct 29 23:21:39.254562 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:21:39.254527 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:21:46.155058 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:46.155017 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-px4td_5f6c3496-906d-4d23-8e0c-a518b01f9ca3/global-pull-secret-syncer/0.log" Oct 29 23:21:46.489309 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:46.489283 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5p8jz_4d78fd0a-b2a6-4688-9435-1e5c972575f8/connection-checker/0.log" Oct 29 23:21:46.609113 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:46.609090 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6vb55_3bf5abc1-44fb-4ec3-b7f7-f9e4b866a09c/connection-checker/0.log" Oct 29 23:21:46.732455 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:46.732435 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vhdkj_60db2db0-73c9-4cb7-aa1b-9c670677bc6e/connection-checker/0.log" Oct 29 23:21:47.212852 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:47.212802 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mmmgd_40badafa-2bba-47cc-b94c-c8f7233c7fb6/konnectivity-agent/0.log" Oct 29 23:21:47.443694 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:47.443660 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-94.ec2.internal_d6ac7ba86bb940ac627127f0b9857b88/haproxy/0.log" Oct 29 23:21:54.866796 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:54.866766 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pw8wr_68657adb-f9ad-417d-a37d-2252e0fecb41/limitador/0.log" Oct 29 23:21:56.260165 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.260123 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787"] Oct 29 23:21:56.260582 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.260568 5073 cpu_manager.go:414] "containerMap: removing container" podUID="622c1f43-c6fd-42b0-8895-cd02f1e7af77" containerName="revoke-keys" Oct 29 23:21:56.260634 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.260583 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622c1f43-c6fd-42b0-8895-cd02f1e7af77" containerName="revoke-keys" Oct 29 23:21:56.260679 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.260667 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="622c1f43-c6fd-42b0-8895-cd02f1e7af77" containerName="revoke-keys" Oct 29 23:21:56.263891 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.263871 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.278971 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.278942 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4b5wq\"/\"default-dockercfg-7vvsf\"" Oct 29 23:21:56.281928 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.281897 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787"] Oct 29 23:21:56.314681 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.314653 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbwq\" (UniqueName: \"kubernetes.io/projected/38fe2da2-f3bc-42b1-911b-3256db7d25a7-kube-api-access-8cbwq\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.314834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.314716 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-sys\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.314834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.314785 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-proc\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.314834 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.314809 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-podres\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.315018 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.314846 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-lib-modules\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.373036 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.373008 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b5wq\"/\"openshift-service-ca.crt\"" Oct 29 23:21:56.373214 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.373116 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b5wq\"/\"kube-root-ca.crt\"" Oct 29 23:21:56.416202 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416176 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbwq\" (UniqueName: \"kubernetes.io/projected/38fe2da2-f3bc-42b1-911b-3256db7d25a7-kube-api-access-8cbwq\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416326 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416226 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-sys\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416326 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416246 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-proc\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416326 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416265 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-podres\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416499 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416322 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-sys\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416499 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416332 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-podres\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416499 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416373 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-proc\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416499 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416418 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-lib-modules\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.416653 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.416547 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38fe2da2-f3bc-42b1-911b-3256db7d25a7-lib-modules\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.436080 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.436056 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbwq\" (UniqueName: \"kubernetes.io/projected/38fe2da2-f3bc-42b1-911b-3256db7d25a7-kube-api-access-8cbwq\") pod \"perf-node-gather-daemonset-m4787\" (UID: \"38fe2da2-f3bc-42b1-911b-3256db7d25a7\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.576036 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.575976 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:56.869942 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:56.869917 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787"] Oct 29 23:21:56.872038 ip-10-0-134-94 kubenswrapper[5073]: W1029 23:21:56.872006 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38fe2da2_f3bc_42b1_911b_3256db7d25a7.slice/crio-5f3f12f1d3edf8bd2106836bbd5aecf8e4f94f3b6696be5adda8e365b39c2ae5 WatchSource:0}: Error finding container 5f3f12f1d3edf8bd2106836bbd5aecf8e4f94f3b6696be5adda8e365b39c2ae5: Status 404 returned error can't find the container with id 5f3f12f1d3edf8bd2106836bbd5aecf8e4f94f3b6696be5adda8e365b39c2ae5 Oct 29 23:21:57.000121 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:57.000089 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" event={"ID":"38fe2da2-f3bc-42b1-911b-3256db7d25a7","Type":"ContainerStarted","Data":"dc364ac7fee3892e8a3a48abd8e0c874b313e433c39aa1d2d8b271f94e1d8122"} Oct 29 23:21:57.000292 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:57.000206 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" event={"ID":"38fe2da2-f3bc-42b1-911b-3256db7d25a7","Type":"ContainerStarted","Data":"5f3f12f1d3edf8bd2106836bbd5aecf8e4f94f3b6696be5adda8e365b39c2ae5"} Oct 29 23:21:57.000292 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:57.000247 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:21:57.041853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:57.041808 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" podStartSLOduration=1.041795144 podStartE2EDuration="1.041795144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:57.040598989 +0000 UTC m=+4032.355993093" watchObservedRunningTime="2026-10-29 23:21:57.041795144 +0000 UTC m=+4032.357189223" Oct 29 23:21:57.277191 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:57.277128 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4n2br_be9c1e3c-0fa7-4509-b4d8-59a57e5fbfd9/cluster-monitoring-operator/0.log" Oct 29 23:21:58.329768 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:58.329716 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8pn2_0a9025bf-fe63-4965-98b5-bace14209a3f/node-exporter/0.log" Oct 29 23:21:58.449439 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:58.449413 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8pn2_0a9025bf-fe63-4965-98b5-bace14209a3f/kube-rbac-proxy/0.log" Oct 29 23:21:58.565853 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:21:58.565829 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8pn2_0a9025bf-fe63-4965-98b5-bace14209a3f/init-textfile/0.log" Oct 29 23:22:00.209693 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:00.209659 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-z9xfl_ac933a85-a255-47fb-8610-fde57045d702/prometheus-operator/0.log" Oct 29 23:22:00.321411 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:00.321384 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-z9xfl_ac933a85-a255-47fb-8610-fde57045d702/kube-rbac-proxy/0.log" Oct 29 23:22:00.442007 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:00.441976 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-h2w6v_2fab7257-c4ac-4f52-8fa6-e6f02b12f9cd/prometheus-operator-admission-webhook/0.log" Oct 29 23:22:02.921408 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:02.921378 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-pcdq8_be7200fc-4e5d-416b-8898-2a8bac5648aa/networking-console-plugin/0.log" Oct 29 23:22:03.016117 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:03.016096 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-m4787" Oct 29 23:22:03.983041 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:03.983012 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/2.log" Oct 29 23:22:03.987227 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:03.987202 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wsjxt_01ed9874-d3cc-4b95-a198-348f68659495/console-operator/3.log" Oct 29 23:22:05.076046 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:05.076015 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-kb25s_0126b13a-114b-474f-8e6d-5142be50e016/download-server/0.log" Oct 29 23:22:06.351683 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:06.351645 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-s4rsk_a1cc6e96-33dc-4727-b69a-c3050aee425c/volume-data-source-validator/0.log" Oct 29 23:22:08.761419 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:08.761373 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-pzncp_0e30adc0-b639-4735-94ab-c80c7d97b200/istio-proxy/0.log" Oct 29 23:22:09.113358 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:09.113279 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gxpdb_edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3/dns/0.log" Oct 29 23:22:09.227559 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:09.227525 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gxpdb_edc1c3f2-5ad8-4864-bedb-ad117d8fc4a3/kube-rbac-proxy/0.log" Oct 29 23:22:09.257449 ip-10-0-134-94 kubenswrapper[5073]: E1029 23:22:09.257411 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p6ccn" containerName="extractor" Oct 29 23:22:09.588286 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:09.588249 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tbx55_340bdb67-9b1d-40f0-8d58-db1157201e45/dns-node-resolver/0.log" Oct 29 23:22:10.680838 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:10.680801 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c9b9fb6c4-hlbln_2368862e-24cf-47f7-8627-ccbe846502d0/registry/0.log" Oct 29 23:22:10.817026 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:10.816995 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cvqq_fbeceea4-8081-4c42-a085-07b5457ed5c7/node-ca/0.log" Oct 29 23:22:11.396513 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:11.396487 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-bb564_2ed97708-4f01-4e75-a35e-9e901a923009/istio-proxy/0.log" Oct 29 23:22:11.700230 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:11.700165 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-kz7rk_818c7229-c7ee-447d-b2fe-c9421d2e0fff/istio-proxy/0.log" Oct 29 23:22:11.761718 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:11.761693 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-lzjlp_9b09d9c3-9695-41d3-bbb7-e7a4f4f87c9a/istio-proxy/0.log" Oct 29 23:22:12.124597 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:12.124568 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b64c8c7d5-7zxkb_7a66a3fc-f006-419b-9442-ef4d8da8bd5e/router/0.log" Oct 29 23:22:12.844270 ip-10-0-134-94 kubenswrapper[5073]: I1029 23:22:12.844237 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-pzncp_0e30adc0-b639-4735-94ab-c80c7d97b200/istio-proxy/0.log"